Skip to content

Module pytools4dart.tools.dbtools

This module contains functions to build properties or prospect database and to extract DART database elements.

Functions

get_models

def get_models(
    dbname,
    search=True
)
Get the list of models of a DART database.

Parameters
----------
dbname: str
    DART database file name or full path
    depending on search argument

search: bool
    If True it will search file name in
    DART_LOCAL/database then in DART_HOME/database
    If False, dbfile is considered as a full path.

Returns
-------
    Pandas DataFrame with model name and model description

Example
-------
.. code-block::

    import pytools4dart as ptd
    dbfile = 'Lambertian_vegetation.db'
    ptd.tools.dbtools.get_models(dbfile)

import2db

def import2db(
    dbFpath,
    name,
    wavelength,
    reflectance,
    direct_transmittance,
    diffuse_transmittance,
    type='Lambertian',
    comments=['# Software: pytools4dart', '# Date: date'],
    verbose=False
)
Add optical properties to a DART database.
DEPRECATED: Use optical_properties_db instead

Parameters
----------
dbFpath: str
    database absolute path
name: str
    name of the new optical properties that it will take in the database.
wavelength: list or np.array
    list of the wavelengths in $\mu m$
reflectance: list or np.array
direct_transmittance: list or np.array
diffuse_transmittance: list or np.array
type: str
    choices: 'Lambertian'
comments: list of strings
verbose: bool

Returns
-------
str
    name of the property as recorded in the database

optical_properties_db

def optical_properties_db(
    db_file,
    name,
    comments='',
    type='lambertian',
    mode='w',
    verbose=False,
    **kwargs
)
Create, append or overwrite DART database with optical properties ((lambertian, hapke or rpv).

Parameters
----------
db_file: str
    Path to the database file.
name: str
    Name of the new optical properties.
comments: str
    Comments relative to the table.
type: str
    Either 'lambertian', 'hapke', or 'rpv'.
mode: str
    Available modes:

        - 'w': Fails if database exist, otherwise write a new database.
        - 'a': appends to existing database otherwise creates it.
        - 'ow': overwrites existing database (removes existing).
verbose: bool
kwargs:
    Depends on type, see Notes.

Returns
-------
str
    name of the property as recorded in the database

Notes
-----
Expeceted arguments in kwargs depend on the optical properties type:

    - lambertian: ['wavelength', 'reflectance', 'direct_transmittance', 'diffuse_transmittance']
    - hapke: ['wavelength', 'w', 'c1', 'c2', 'c3', 'c4', 'h1', 'h2']
    - rpv: ['wavelength', 'reflectance', 'k', 'g', 'h']

These arguments are expected to be lists or numpy 1D arrays with the same length (or 1 value for automatic filling)

WARINING: wavelengths are expeceted in micrometers (not nanometers)

Examples
--------
>>> import pytools4dart as ptd
>>> import pandas as pd
>>> import sqlite3
>>> from pytools4dart.tools.dbtools import optical_properties_db, search_dbfile, get_models

# copy a lambertian property into new database
>>> dart_db_file = search_dbfile('Lambertian_vegetation.db')
>>> conn = sqlite3.connect(dart_db_file)
>>> name = 'acer_alnus_fraxinus_tilia_wood'
>>> models = pd.read_sql('select * from _comments', conn)
>>> comments = models.loc[models.model == name].Comments.iloc[0]
>>> data = pd.read_sql('select * from {}'.format(name), conn).drop('Id', axis=1)
>>> new_db_file = ptd.getdartenv()['DART_LOCAL'] / 'database' / 'test.db'
>>> optical_properties_db(new_db_file, name, **data.to_dict(), comments=comments, mode='ow')
'acer_alnus_fraxinus_tilia_wood'

# add a new lambertian property
>>> name = 'test spectrum'
>>> wavelength = [1, 2, 3]
>>> reflectance = [.1, .2, .3]
>>> direct_transmittance = [0, 0, 0]
>>> diffuse_transmittance = [.9, .8, .7]
>>> optical_properties_db(new_db_file, name=name,                          wavelength=wavelength, reflectance=reflectance,                          direct_transmittance=direct_transmittance,                          diffuse_transmittance=diffuse_transmittance,                          comments = ["Species: test",                                      "Date: 2020",                                      "Project: pytools4dart"],                          mode='a')
'test_spectrum'

# list models of the new database
>>> get_models(new_db_file, search=False).model # doctest: +NORMALIZE_WHITESPACE
    0    acer_alnus_fraxinus_tilia_wood
    1                     test_spectrum
    Name: model, dtype: object

prospect_db

def prospect_db(
    db_file,
    N=1.8,
    Cab=30,
    Car=10,
    CBrown=0,
    Cw=0.012,
    Cm=0.01,
    Can=0,
    prospect_version='D',
    mode='w',
    inmem=True,
    verbose=False
)
Create or append properties and corresponding spectra to a DART prospect database.

It is 100x faster then if computed within DART (~11ms/entry), but it is limited to prospect at the moment.
Duplicates and already existing entries are internally removed from computation.
Open an issue if fluorescence/fluspect is needed, we'll develop it on demand.

Parameters
----------
db_file: str
    Path to database file.

N: float
    Messophyl structural parameter [1.0-3.5].

Cab: float
    Chlorophyll content [ug cm^-2].

Car: float
    Carotenoid content [ug cm^-2].

CBrown: float
    Fraction of senescent matter [0-1].

Cw: float
    Water column [cm].

Cm: float
    Dry matter content [g cm^-2].

Can: float
    Anthocyanin content [ug cm^-2].

prospect_version: str
    'D' or '5'

mode: str
    Available modes:

        - 'w': Fails if database exist, otherwise write a new database.
        - 'a': appends to existing database otherwise creates it.
        - 'ow': overwrite existing database (removes existing).

inmem: bool
    If True, it loads existing database in memory, inserts all properties/spectra and copies it back to disk.
    10 000 entries takes about 750 MB memory and 1min50s to compute.

    If 'False', the operations are made directly on the database file, which is 1.5x slower compared to in memory.
    This option should be set to 'False' if the number of properties is very large (>10 000) or memory is small.
    Or it should be split in several database.

verbose: bool
    Print messages if True.

Returns
-------
pandas.DataFrame
    Table of properties with corresponding model name and prospect file,
    ready to fill DART simulation parameters.

Examples
--------
# Create a new database named 'prospect_test.db' and fill it with 100 properties.
>>> import pytools4dart as ptd
>>> import pandas as pd
>>> import numpy as np
>>> size = 100
>>> np.random.seed(0)

>>> user_data = ptd.getdartenv()['DART_LOCAL']
>>> db_file = user_data / 'database' / 'prospect_test.db'

>>> properties = pd.DataFrame({'N':np.random.uniform(1,3,size), 'Cab':np.random.uniform(0,30,size),                   'Car':np.random.uniform(0,5,size), 'Can':np.random.uniform(0,2,size)})
>>> prop_table = ptd.dbtools.prospect_db(db_file, **properties.to_dict('list'), mode='ow')
>>> prop_table.drop(['prospect_file', 'file_hash'], axis=1) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
           N        Cab       Car  ...    V2Z  prospect_version          model
0   2.097627  20.334496  1.558979  ... -999.0                 D   hc2063992474
1   2.430379   8.100239  3.481717  ... -999.0                 D   hc1313623719
2   2.205527  22.055821  1.888759  ... -999.0                 D   hc1123283726
3   2.089766  28.865656  0.898018  ... -999.0                 D   hc1147596031
4   1.847310   7.462594  0.123394  ... -999.0                 D  hcm1082664945
..       ...        ...       ...  ...    ...               ...            ...
95  1.366383  14.713764  1.121585  ... -999.0                 D    hc240033769
96  2.173026   6.822439  0.489222  ... -999.0                 D   hc1557782782
97  1.040215   7.630694  4.310958  ... -999.0                 D    hc848330328
98  2.657880   1.740875  4.864597  ... -999.0                 D     hc49086653
99  1.009391  13.032499  4.804173  ... -999.0                 D  hcm1612636386
<BLANKLINE>
[100 rows x 12 columns]

# Add 100 properties more to the same database.
>>> properties = pd.DataFrame({'N':np.random.uniform(1,3,size), 'Cab':np.random.uniform(0,30,size),                   'Car':np.random.uniform(0,5,size), 'Can':np.random.uniform(0,2,size)})
>>> prop_table = ptd.dbtools.prospect_db(db_file, **properties.to_dict('list'), mode='a')

search_dbfile

def search_dbfile(
    dbname='Lambertian_vegetation.db'
)
Search for database file in DART/database or DART/user_data/database

Parameters
----------
dbname: str
    Database file name

Returns
-------
str
    full path of database if file exist