he_the_great (he_the_great) wrote,
he_the_great
he_the_great

Parsing CSV Files that exceed Available Memory

It is not to unusual to read very large files and D does have a few options one is std.stdio.File. But using std.csv requires a input range of dchar which isn't available in the standard library (that I know of) for files.

I'd done some previous work in providing a file as range in my OpenStreetMap project. I had to make a few modifications so that it would provide an input range of dchar. While I tried to make it able to read files encode in UTF-8, UTF-16, or UTF-32, my plans were not completed since FileRange would need to read multiple bytes for those not UTF-8.

If someone is more interested then it should be possible to remove the cast on line 70 and make a conditional which correctly builds the character size desired before returning it.
Tags: dlang
Subscribe

  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 2 comments