Setting csv arguments ',' in importer without 'source'

[Notmarrco]

Hi,

I have a csv with ',' separators instead of ';'.
In my importer I thought I could use :

preconditions:
    format: { type: csv, arguments: [','] }

but it's not possible.

I saw in the Configuration dependencyInjection that the arguments are available only in source or target options. Then when I try with source, the separator is well set, but the source: uri is mandatory.
So I'm left with :

• either I can choose my file in command line, but I can't use ',' separator
• or I have to always use the same name, defined in the source options

Am I missing something ?

Thx

[thim81]

Hi Notmarrco,

Could you share your full configuration?

The preconditions part should be a valid format

 preconditions:
                format: csv               #format of data must be csv OR excel OR xml
                fieldcount: 2               #must have this number of fields
....

With regards to the CSV "," vs ";" I think the CSV format handler detects the seperator itself, so it is not necessary to define it, but I'm not 100% since in my use-cases I have always used Excel documents.

[Notmarrco]

Hi

Here is my configuration :

    importers:
        generic_importer:
            #some context information that is passed through the whole process
            context:
                key: generic_importer
            #automaticly recognize this importer by meeting of the conditions below
            preconditions:
                format: csv
                fieldcount: 3 
                fields:
                    - 'fullname'
                    - 'login'
                    - 'email'
                filename: 'Export_generic.csv'

            source: 
                type: file
                uri: web/uploads/tmp/Export_generic.csv
                format: { type: csv, arguments: [','] }                         

            #target of import
            target:
                type: service
                service: 'generic.user.crud' 
                method: processImportRow  

Here the problem is that my csv file is separated with ',' and not with semicolons. So without the source part of my configuration (this one : )

            source: 
                type: file
                uri: web/uploads/tmp/Export_generic.csv
                format: { type: csv, arguments: [','] }           

when I try to import a file, it does find only one big field with "fullname,login,email" as header, instead of my 3 fields.

The only way I found was to use the "source" part, but :

• I can't modify the name of the file I want to import in cli
• and this source configuration takes precedence over preconfigurations parameters.

What I want is to make rid of "source" configuration part and to be able to say to the importer that the separator is ','.

[thim81]

Hi Notmarco,

I haven't run into this issue, but I assume indeed that this is linked to the deboer bundle and how it's implemented.

Perhaps Markus know how to solve this.

[Notmarrco]

Hi, thanks for your answer. I will wait for @mathielen then.

I also think it can be related to the ImportEngine, and the auto csv class : it seems that it find a "space" in the file and use it as a separator when guessing what is the separator.

In fact it shouldn't have to "guess" the separator, since I gave him the fields (and I also tried with a fieldset) I'm waiting for..

[mathielen]

Hi Notmarco,

Ok so, its perfectly correct to do

            source: 
                type: file
                uri: web/uploads/tmp/Export_generic.csv
                format: { type: csv, arguments: [','] }      

And it works - for exactly one specific file. You are right, you cannot invoke the importer from the CLI then. The importer would be very "static".

This is what the *StorageProvider classes are for. So by default there is a FileStorageProvider which uses a FormatDiscoverLocalFileStorageFactory to automatically find the format of the file, which in your case does not need to be discovered.
So what you could do is, register a different source_provider and tell the engine to use it instead. (Its the second argument of the CLI command). You could use a ServiceStorageProvider (type=service) and handle your logic there.
And yes, I know its not very elegant. I will put this on my roadmap :)

Best
Markus

[thim81]

+1 as new feature

[Notmarrco]

Hi,

Any update on this ? I would really like to avoid using the "source" field and only use the auto discovering of files using the "preconditions" if I can give it the good value separator.

thx for the bundle anyway :)

[Notmarrco]

Hello again,

I'm willing to fix this, but I'm not sure what is the right way to do it.

Until now I just uploaded files with the same name, but I want to change it.

I'm actually quite confused with all the "factories", "providers" and so. 😄

If I understand correctly, my problem is that in ImporterPrecondition, when I test the fields, I have $anyFields (that is $storage->getFields()) returning all fields in one big field because the $storage couldn't find the right separator.

But what I don't understand is that my sourceStorage (ie LocalFileStorage) is working ok and the format={ type: csv, arguments: [','] } is the right one, so the precondition should test the fields given this format. And I can't find why it is happening...

@mathielen if you can point me in the right direction, I will try to make a PR.

Thank you

[Notmarrco]

Ok I was totally wrong before :)

it works when I have a source storage defined, because then it uses the getFields() with the right format, but I can't have a source storage without an uri, so that's why @mathielen suggested I should use a StorageProvider that can guess the right separator.

And now I find that the "directory" storage_provider that I am using, has MimeType discovery and also csv auto delimiter : https://github.com/mathielen/ImportEngineBundle/blob/master/DependencyInjection/MathielenImportEngineExtension.php#L286

So I tried just that and it worked like a charm ... So my problem was in fact that the auto delimiter was working properly but it found "_" instead of "," for my delimiter, because my fields names had underscores... I changed that and all is working well.

At the end here are the possible fixes :

• change the importance order of the delimiters (like "," is more likely to happen than "_") not so good idea
• or pass the delimiter to the directory provider in the first place

I'm not sure it's even worth to bother ^^