Module pytools4dart.tools.dbtools
This module contains functions to build properties or prospect database and to extract DART database elements.
Functions
get_models
def get_models(
dbname,
search=True
)
Get the list of models of a DART database.
Parameters
----------
dbname: str
DART database file name or full path
depending on search argument
search: bool
If True it will search file name in
DART_LOCAL/database then in DART_HOME/database
If False, dbfile is considered as a full path.
Returns
-------
Pandas DataFrame with model name and model description
Example
-------
.. code-block::
import pytools4dart as ptd
dbfile = 'Lambertian_vegetation.db'
ptd.tools.dbtools.get_models(dbfile)
import2db
def import2db(
dbFpath,
name,
wavelength,
reflectance,
direct_transmittance,
diffuse_transmittance,
type='Lambertian',
comments=['# Software: pytools4dart', '# Date: date'],
verbose=False
)
Add optical properties to a DART database.
DEPRECATED: Use optical_properties_db instead
Parameters
----------
dbFpath: str
database absolute path
name: str
name of the new optical properties that it will take in the database.
wavelength: list or np.array
list of the wavelengths in $\mu m$
reflectance: list or np.array
direct_transmittance: list or np.array
diffuse_transmittance: list or np.array
type: str
choices: 'Lambertian'
comments: list of strings
verbose: bool
Returns
-------
str
name of the property as recorded in the database
optical_properties_db
def optical_properties_db(
db_file,
name,
comments='',
type='lambertian',
mode='w',
verbose=False,
**kwargs
)
Create, append or overwrite DART database with optical properties ((lambertian, hapke or rpv).
Parameters
----------
db_file: str
Path to the database file.
name: str
Name of the new optical properties.
comments: str
Comments relative to the table.
type: str
Either 'lambertian', 'hapke', or 'rpv'.
mode: str
Available modes:
- 'w': Fails if database exist, otherwise write a new database.
- 'a': appends to existing database otherwise creates it.
- 'ow': overwrites existing database (removes existing).
verbose: bool
kwargs:
Depends on type, see Notes.
Returns
-------
str
name of the property as recorded in the database
Notes
-----
Expeceted arguments in kwargs depend on the optical properties type:
- lambertian: ['wavelength', 'reflectance', 'direct_transmittance', 'diffuse_transmittance']
- hapke: ['wavelength', 'w', 'c1', 'c2', 'c3', 'c4', 'h1', 'h2']
- rpv: ['wavelength', 'reflectance', 'k', 'g', 'h']
These arguments are expected to be lists or numpy 1D arrays with the same length (or 1 value for automatic filling)
WARINING: wavelengths are expeceted in micrometers (not nanometers)
Examples
--------
>>> import pytools4dart as ptd
>>> import pandas as pd
>>> import sqlite3
>>> from pytools4dart.tools.dbtools import optical_properties_db, search_dbfile, get_models
# copy a lambertian property into new database
>>> dart_db_file = search_dbfile('Lambertian_vegetation.db')
>>> conn = sqlite3.connect(dart_db_file)
>>> name = 'acer_alnus_fraxinus_tilia_wood'
>>> models = pd.read_sql('select * from _comments', conn)
>>> comments = models.loc[models.model == name].Comments.iloc[0]
>>> data = pd.read_sql('select * from {}'.format(name), conn).drop('Id', axis=1)
>>> new_db_file = ptd.getdartenv()['DART_LOCAL'] / 'database' / 'test.db'
>>> optical_properties_db(new_db_file, name, **data.to_dict(), comments=comments, mode='ow')
'acer_alnus_fraxinus_tilia_wood'
# add a new lambertian property
>>> name = 'test spectrum'
>>> wavelength = [1, 2, 3]
>>> reflectance = [.1, .2, .3]
>>> direct_transmittance = [0, 0, 0]
>>> diffuse_transmittance = [.9, .8, .7]
>>> optical_properties_db(new_db_file, name=name, wavelength=wavelength, reflectance=reflectance, direct_transmittance=direct_transmittance, diffuse_transmittance=diffuse_transmittance, comments = ["Species: test", "Date: 2020", "Project: pytools4dart"], mode='a')
'test_spectrum'
# list models of the new database
>>> get_models(new_db_file, search=False).model # doctest: +NORMALIZE_WHITESPACE
0 acer_alnus_fraxinus_tilia_wood
1 test_spectrum
Name: model, dtype: object
prospect_db
def prospect_db(
db_file,
N=1.8,
Cab=30,
Car=10,
CBrown=0,
Cw=0.012,
Cm=0.01,
Can=0,
prospect_version='D',
mode='w',
inmem=True,
verbose=False
)
Create or append properties and corresponding spectra to a DART prospect database.
It is 100x faster then if computed within DART (~11ms/entry), but it is limited to prospect at the moment.
Duplicates and already existing entries are internally removed from computation.
Open an issue if fluorescence/fluspect is needed, we'll develop it on demand.
Parameters
----------
db_file: str
Path to database file.
N: float
Messophyl structural parameter [1.0-3.5].
Cab: float
Chlorophyll content [ug cm^-2].
Car: float
Carotenoid content [ug cm^-2].
CBrown: float
Fraction of senescent matter [0-1].
Cw: float
Water column [cm].
Cm: float
Dry matter content [g cm^-2].
Can: float
Anthocyanin content [ug cm^-2].
prospect_version: str
'D' or '5'
mode: str
Available modes:
- 'w': Fails if database exist, otherwise write a new database.
- 'a': appends to existing database otherwise creates it.
- 'ow': overwrite existing database (removes existing).
inmem: bool
If True, it loads existing database in memory, inserts all properties/spectra and copies it back to disk.
10 000 entries takes about 750 MB memory and 1min50s to compute.
If 'False', the operations are made directly on the database file, which is 1.5x slower compared to in memory.
This option should be set to 'False' if the number of properties is very large (>10 000) or memory is small.
Or it should be split in several database.
verbose: bool
Print messages if True.
Returns
-------
pandas.DataFrame
Table of properties with corresponding model name and prospect file,
ready to fill DART simulation parameters.
Examples
--------
# Create a new database named 'prospect_test.db' and fill it with 100 properties.
>>> import pytools4dart as ptd
>>> import pandas as pd
>>> import numpy as np
>>> size = 100
>>> np.random.seed(0)
>>> user_data = ptd.getdartenv()['DART_LOCAL']
>>> db_file = user_data / 'database' / 'prospect_test.db'
>>> properties = pd.DataFrame({'N':np.random.uniform(1,3,size), 'Cab':np.random.uniform(0,30,size), 'Car':np.random.uniform(0,5,size), 'Can':np.random.uniform(0,2,size)})
>>> prop_table = ptd.dbtools.prospect_db(db_file, **properties.to_dict('list'), mode='ow')
>>> prop_table.drop(['prospect_file', 'file_hash'], axis=1) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
N Cab Car ... V2Z prospect_version model
0 2.097627 20.334496 1.558979 ... -999.0 D hc2063992474
1 2.430379 8.100239 3.481717 ... -999.0 D hc1313623719
2 2.205527 22.055821 1.888759 ... -999.0 D hc1123283726
3 2.089766 28.865656 0.898018 ... -999.0 D hc1147596031
4 1.847310 7.462594 0.123394 ... -999.0 D hcm1082664945
.. ... ... ... ... ... ... ...
95 1.366383 14.713764 1.121585 ... -999.0 D hc240033769
96 2.173026 6.822439 0.489222 ... -999.0 D hc1557782782
97 1.040215 7.630694 4.310958 ... -999.0 D hc848330328
98 2.657880 1.740875 4.864597 ... -999.0 D hc49086653
99 1.009391 13.032499 4.804173 ... -999.0 D hcm1612636386
<BLANKLINE>
[100 rows x 12 columns]
# Add 100 properties more to the same database.
>>> properties = pd.DataFrame({'N':np.random.uniform(1,3,size), 'Cab':np.random.uniform(0,30,size), 'Car':np.random.uniform(0,5,size), 'Can':np.random.uniform(0,2,size)})
>>> prop_table = ptd.dbtools.prospect_db(db_file, **properties.to_dict('list'), mode='a')
search_dbfile
def search_dbfile(
dbname='Lambertian_vegetation.db'
)
Search for database file in DART/database or DART/user_data/database
Parameters
----------
dbname: str
Database file name
Returns
-------
str
full path of database if file exist