Clojure Q&A - Recent questions in data.csv

Supporting more malformed files in clojure.data.csv/read-csv

Mon, 19 Apr 2021 21:22:10 +0000

Imagine we have the following CSV file

A,B,C
this is,"a badly" quoted, file

When trying to parse this file with clojure.data.csv/read-csv, I get the following exception

{:type java.lang.Exception
 :message "CSV error (unexpected character:  )"
 :at [clojure.data.csv$read_quoted_cell invokeStatic "csv.clj" 37]}

This file is clearly malformed, but I've seen a file like this in the wild so it would be nice if read-csv handled extra content after the quoted portion by parsing this to

["this is" "a badly quoted" " file"]

Potential problem with this proposal:

If there's a separator inside the quotes this becomes harder to interpret. e.g.

this is,"a, badly" quoted, file

could be parsed to

["this is" "a, badly quoted " " file"]

["this is" "\"a" " badly\" quoted " " file"]

While this second interpretation seems improbable to me, I'm not sure what the "best effort" interpretation strategy is in this case

Support CSV Injection Escape Mechanisms

Fri, 23 Feb 2018 17:50:58 +0000

CSVs generated using clojure.csv is succeptible to injection attacks. It'd be a nice enhancement to have an option to apply this transformation on behalf of users.

For example:

(with-open (link: writer (io/writer "out-file.csv"))
(csv/write-csv writer

             (link: ["abc" "def")
              (link: "ghi" "=jkl")]))

See https://www.owasp.org/index.php/CSV_Injection

Adding namespace qualifiers in documentation

Sat, 11 Nov 2017 12:49:29 +0000

README.md has code examples that are missing namespace qualifiers, Adding them will make it easy to copy the code in REPL and make it work.
In addition to that there are couple of fixes for misspelled words.
The commit is in the following fork for reference

https://raw.githubusercontent.com/as17237/data.csv/master/README.md

Use Reducers/Transducers for better performance & resource handling

Thu, 15 Sep 2016 23:04:43 +0000

One problem when using the clojure.data.csv library is that it's built upon lazy sequences which can lead to inefficiencies when processing large amounts of data, for example even before any transformation is done the base-line parsing of 1gb of data of CSV takes about 50s on my machine. Other parsers available on the JVM can parse this quantity of data in less than 4 seconds.

I'd like to discuss how we might port clojure.data.csv to use a reducer/transducer model, for improved performance and resource handling. Broadly speaking I think there are a few options:

Implement this as a secondary alternative API in c.d.csv leaving the existing API and implementation as is for legacy users.
Replace the API entirely with no attempt at retaining backwards compatibility.
Retain the same public API contracts, whilst trying to reimplement it underneath in terms of reducers/transducers. Use transducers underneath but use sequence to retain the current parse-csv lazy-seq contract, whilst offering access into a new pure transducer/reducer based API for non legacy users or those who don't require a lazy-seq based implementation.

1 and 3 are essentially the same idea, except in 3 users get the benefit of a faster underlying implementation, there may also be other options.

I think 3, if possible, would be the best option.

Options 1 and 2 raise the question, of making no attempt at backwards compatibility or improving the experience for legacy users.

Before delving into the details of how the reducer/transducer implementation, I'm curious what the core team think of exploring
this further.

Double quote at beginning of cell throws exception

Fri, 05 Aug 2016 02:34:06 +0000

If a cell has a double quote (an escaped quotation mark) as the first characters in the cell then an exception is thrown.

For example:
(csv/read-csv "this,\"\"that\"\",the other")

produces:
{color:red}
Exception CSV error (unexpected character: t) clojure.data.csv/read-quoted-cell (csv.clj:36)
{color}

but this:
(csv/read-csv "this, \"\"that\"\",the other")

produces this correct output:
((link: "this" " \"\"that\"\"" "the other"))

Port data.csv to clojurescript

Mon, 11 Apr 2016 10:53:02 +0000

Make data.csv available for clojurescript users.

Add project.clj for easier local development

Thu, 07 Apr 2016 09:59:35 +0000

Add project.clj for easier local development

Specify RFC4180 compatibilty in README

Wed, 18 Mar 2015 13:09:42 +0000

In the README it says: "Follows the RFC4180 specification but is more relaxed."
This is an oxymoron and confusing in other regards. E.g.:
- What does "relaxed" mean?
- If it is more "relaxed" than the specification, how can it follow it?
- Does it follow the specification, or only parts of it?

Problem: If I use this lib to generate CSV for a third party, can I say "This is RFC4180 conform CSV" and feel safe with it? Or should I add "but it is more relaxed" :)

The task could be to add more specific explanation or a comparison table if necessary.

write-csv and quote? predicate

Tue, 21 Oct 2014 09:02:37 +0000

In version 0.1.2 the quote? predicate is called after the object to be written into a cell is converted into a string (see line 99). If the predicate quote? would be applied to the object instead, function write-csv could be called as follows:

(write-csv
"test.csv"
(link: [1 "text") (link: 2 "text")]
:quote string?)

In the current version every cell value is a string.

Allow read-csv to read files without quoting.

Thu, 29 May 2014 15:14:02 +0000

I would like to be able to read file with the following format:
- '|' separated
- Unquoted.. eg. \" can appear in the strings, in particular

at the beginning, and not at the end.

I need to set a nul quote character, but this doesn't currently work.
The following is a workaround, where a '.' is unlikely to appear in first
character of the sting.

(csv/read-csv in-file :separator \| :quote .))

I would like to be able to be explicit:

(csv/read-csv in-file :separator \| :quote nul))

read-csv can not handle white-space at end of line

Fri, 24 May 2013 18:50:04 +0000

When whitespace is present after the closing \" the clojure reader crashes with a weird error.
It took me some time to notice it was a white-space issue as whitespace is .... not visible.

See an example of the error below.

=> (read-csv (java.io.StringReader. "\"a\" " ))
Exception CSV error (unexpected character: ) clojure.data.csv/read-quoted-cell (csv.clj:36)
=> (read-csv (java.io.StringReader. "\"a\"" ))
((link: "a"))

pom.xml directives

Fri, 10 Feb 2012 10:59:36 +0000

If you build data.csv alone with the current pom.xml you get a couple of warnings and test are not executed. With the recent versions of Maven, these warnings can break the build.

A fixed (I hope!) version is attached.