Metadata-Version: 2.1
Name: lm-dataformat
Version: 0.0.9
Summary: A utility for storing and reading files for LM training.
Home-page: https://github.com/leogao2/lm_dataformat
Author: Leo Gao
Author-email: leogao31@gmail.com
License: UNKNOWN
Description: # LM_Dataformat
        
        Utilities for storing data for LM training.
        
        
        ## Basic Usage
        
        To write:
        
        ```
        ar = Archive('output_dir')
        
        for x in something():
          # do other stuff
          ar.add_data(somedocument, meta={
            'example': stuff,
            'someothermetadata': [othermetadata, otherrandomstuff],
            'otherotherstuff': True
          })
        
        # remember to commit at the end!
        ar.commit()
        ```
        
        To read:
        
        ```
        rdr = Reader('input_dir_or_file')
        
        for doc in ar.stream_data():
          # do something with the document
        ```
        
Platform: UNKNOWN
Requires-Python: >=3.6
Description-Content-Type: text/markdown
