Local Convolutional features with Unsupervised Training for Image Retrieval


Project page





Dataset


Rome Patches is a dataset that allows to jointly evaluate performance of a local patch descriptor on a patch retrieval task and on image retrieval.

If you use the dataset, please cite the following papers:

Bibtex:

@inproceedings{paulin2015local,
  • title={Local Convolutional Features with Unsupervised Training for Image Retrieval},
  • author = {Mattis Paulin and Matthijs Douze and Zaid Harchaoui and Julien Mairal and Florent Perronnin and Cordelia Schmid},
  • year={2015},
  • booktitle = {International Conference on Computer Vision (ICCV)}
}

@inproceedings{li2010location,
  • title={Location Recognition using Prioritized Feature Matching},
  • author = {Yunpeng Li and Noah Snavely and Dan Huttenlocher},
  • year={2010},
  • booktitle = {European Conference on Computer Vision (ECCV)}
}

Files

Patches

Images


How to use

Patches

  • Get all data files by either clicking on previous links or running the following script 
  wget http://pascal.inrialpes.fr/data2/paulin/RomePatches/patch_retrieval_Rome_train.mat
  wget http://pascal.inrialpes.fr/data2/paulin/RomePatches/patch_retrieval_Rome_train_labels.mat
  wget http://pascal.inrialpes.fr/data2/paulin/RomePatches/patch_retrieval_Rome_test.mat
  wget http://pascal.inrialpes.fr/data2/paulin/RomePatches/patch_retrieval_Rome_test_labels.mat
  wget http://pascal.inrialpes.fr/data2/paulin/RomePatches/patch_retrieval_Rome_sifts_train.mat
  wget http://pascal.inrialpes.fr/data2/paulin/RomePatches/patch_retrieval_Rome_sifts_test.mat
  • Get the testing script here or: 
  wget http://lear.inrialpes.fr/people/paulin/projects/RomePatches/ckn_patch_retrieval.m
  • Try out the test scripts on SIFT (first argument is the train/test split, second is verbosity and third triggers SIFT baseline evaluation): 
  matlab -nodisplay -r "ckn_patch_retrieval('train', false, true), exit"
  matlab -nodisplay -r "ckn_patch_retrieval('test', false, true), exit" 
Elapsed time is 0.961501 seconds.

ans =

    0.9164
Elapsed time is 0.943030 seconds.

ans =

    0.8787
  • Modify the test script here to use your patch descriptor implementation: 
 
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    %%% CHANGE THIS LINE TO YOUR PATCH DESCRIPTOR IMPLEMENTATION  %%
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

	    descs = encode(Xtr);
	    
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
  • Run the evaluation 
  matlab -nodisplay -r "ckn_patch_retrieval('train', false, false), exit"
  matlab -nodisplay -r "ckn_patch_retrieval('test', false, false), exit"

Images

  • Go to the Rome16K dataset webpage and download it. You need to manually download images from their urls.
  • Use the .txt files above to select the 2x2000 images (train + test, 1000 queries and 1000 targets) needed for the dataset. In each file lies a list of image name followed by its label.
  • Encode the images using your own descriptors.
  • Evaluate. Standard measure is mAP.




CKN Code


This is the code designed to replicate our results with CKNs. The version we make available relies on J. Mairal's code available here. If you use this code, we request you cite the following papers:

Bibtex:

@inproceedings{paulin2015local,
  • title={Local Convolutional Features with Unsupervised Training for Image Retrieval},
  • author = {Mattis Paulin and Matthijs Douze and Zaid Harchaoui and Julien Mairal and Florent Perronnin and Cordelia Schmid},
  • year={2015},
  • booktitle = {International Conference on Computer Vision (ICCV)}
}

@inproceedings{mairal2014convolutional,
  • title={Convolutional Kernel Networks},
  • author = {Julien Mairal and Piotr Koniusz and Zaid Harchaoui and Cordelia Schmid},
  • year={2014},
  • booktitle = {Advances in Neural Information Processing Systems (NIPS)}
}

The code is pure matlab (with mex) and sould therefore run on all platforms; it has however only been tested on a Linux x86_64 architecture. In the paper, we used a different, faster version, and there are some slight differences in results because of rounding errors. We plan to make the faster version available very soon.

Files

How to use

The code heavily relies on J. Mairal's package, which you need to download and compile: 

 wget -O ckn-matlab.tar.gz http://ckn.gforge.inria.fr/hitcounter2.php?file=34280/ckn-matlab-v1.0-svn2014-11-19.tar.gz
 tar xzfv ckn-matlab.tar.gz
 wget -O spams-matlab.tar.gz http://spams-devel.gforge.inria.fr/hitcounter2.php?file=33814/spams-matlab-v2.5-svn2014-07-04.tar.gz
 tar xzfv spams-matlab.tar.gz
 cd spams-matlab 
 matlab -nodisplay -r "compile; exit"
 cd ..

Because of the new formulation we derived (see Sec. 4.2), the extraction code is not compatible with the parameters we provide. This can however be solved by changing the encode_layer.m file by replacing the following lines:

   
% convolution with filters 

   outmap=model.Z' * X;
   
% nonlinearities 

   outmap=ones(size(model.Z,2),1)*sum(X.^2) + sum(model.Z.^2)' * ones(1,size(X,2))  -2*outmap;
   outmap=exp(-outmap/(model.sigma*model.sigma));
   outmap=diag(sqrt(model.w))*outmap;
   nchannels=size(outmap,1);
   outmap=bsxfun(@times,outmap,nrm);
   outmap=outmap';
   outmap = reshape(outmap,[sx sx nchannels]);

by: 

   
% convolution with filters 

   outmap=bsxfun(@plus, model.W' * X, model.b);
   
% nonlinearities

   outmap=exp(outmap);
   nchannels=size(outmap,1);
   outmap=bsxfun(@times,outmap,nrm);
   outmap=outmap';
   outmap = reshape(outmap,[sx sx nchannels]);

Alternatively, you can go to the ckn folder and download the new encode_layer.m: 

 cd ckn
 wget -O encode_layer.m http://lear.inrialpes.fr/people/paulin/projects/RomePatches/encode_layer.m
 cd ..

To use in the previous script, you can for instance change the required lines to 

    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    %%% CHANGE THIS LINE TO YOUR PATCH DESCRIPTOR IMPLEMENTATION  %%
    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
	  

    addpath('ckn')
    addpath('spams-matlab/build')
    load('ckn-grad.mat')
    descs = encode(Xtr, model);
	    

    %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

The following code should give the expected result (note that you might have to change the path to your gcc lib for the preload. This step is required by the SPAMS package): 

  wget http://pascal.inrialpes.fr/data2/paulin/RomePatches/ckn-grad.mat
  export LIB_GCC=/usr/lib/gcc/x86_64-redhat-linux/4.9.2/
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/gcc/x86_64-redhat-linux/4.9.2/:/
  export LD_PRELOAD=$LIB_GCC/libgfortran.so:$LIB_GCC/libgcc_s.so:$LIB_GCC/libstdc++.so:$LIB_GCC/libgomp.so

  matlab -nodisplay -r "ckn_patch_retrieval('train', 0,0), exit"

Result should be 

ans =
  0.9168

Feedback is welcome at mattis dot paulin at inria dot fr.