# Carpyncho - VVV Features Dataset

- Authors: Cabral, J.B.
- Contact: jbcabral@unc.edu.ar
- Pub-Date: 2022-05-25
- Last modification: 2022-05-25

## Description

Carpyncho, is a catalog  that we hope will be reutilized to search for and
characterize time variable data of the ~PiB size VVV/VVVx survey.
Is being developed for the detection and classification of periodic variables.
For this purpose the stacked pawprint data from the VDFS CASU v >= 1.3 catalogs
have been crossed matched with the VDFS CASU v1.3 tile catalogs into a
Parquet files. 

## Reference


> Cabral, J. B., Ramos, F., Gurovich, S., & Granitto, P. (2020).
> Automatic Catalog of RRLyrae from ∼ 14 million VVV Light Curves:
> How far can we go with traditional machine-learning?
> https://arxiv.org/abs/2005.00220

Bibtex entry

```bib
@ARTICLE{2020arXiv200500220C,
       author = {{Cabral}, Juan B. and {Ramos}, Felipe and {Gurovich}, Sebasti{\'a}n and
         {Granitto}, Pablo},
        title = "{Automatic Catalog of RRLyrae from $\sim$ 14 million VVV Light Curves: How far can we go with traditional machine-learning?}",
      journal = {arXiv e-prints},
     keywords = {Astrophysics - Instrumentation and Methods for Astrophysics, Astrophysics - Solar and Stellar Astrophysics, Computer Science - Machine Learning, Statistics - Machine Learning},
         year = 2020,
        month = may,
          eid = {arXiv:2005.00220},
        pages = {arXiv:2005.00220},
archivePrefix = {arXiv},
       eprint = {2005.00220},
 primaryClass = {astro-ph.IM},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2020arXiv200500220C},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}
```


## Tutorial

First of all: ignore the *_test* folder.

This dataset contains three types of catalogs:

1. Light-Curves catalogs tutorial (lc, inside `lcurves/`)
2. Features catalogs tutorial (features, inside `lcurves/`)
3. Carpyncho RR-Lyrae V.1.0 Catalogs(cp_rr_v1, inside `catalogs/`)

The tutorial to understand each of these types of catalogs can be found
[here](https://carpyncho-py.readthedocs.io/en/latest/tutorials/catalogs/catalogs.html)

All the files are stored compresed in bzip2 format and need to be uncompressed.
This can be achieved with many visual compressors.
From command line (in most linux and osx) you can run:

```console
$ bzip2 -k filename.parquet.bz2
```

The uncompressed file is in `parquet` format.

Apache Parquet is a free and open-source column-oriented data storage format.
It provides efficient data compression and encoding schemes with enhanced
performance to handle complex data in bulk.


**Python**

The easy way is to use the [Carpyncho Python client](https://github.com/carpyncho/carpyncho-py),
but if you wan't to manually download and manipulate the files you need to
first install pandas and pyarrow with

```console
$ pip install pandas pyarrow
```

and then from the Python code/terminal

```python

# load the library into memory
>>> import pandas as pd

# read the parquet file into a dataframe
>>> pd.read_parquet("filename.parquet")
```

Check for more info
[here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html)

**R**

```R
# install the arrow library (only once)
> install.packages("arrow", repos = "https://dl.bintray.com/ursalabs/arrow-r")

# load the library into memory
> library("arrow")

# read the parquet file into a dataframe
> df = read_parquet("filename.parquet", as_data_frame=TRUE)
```