Upload Image: MD5Sum problem

flyingdutchie · November 17, 2007

When I upload an image, using Kallasoft Smugmug API, i provide the byte-array and its MD5 sum.

First i tried to upload it without the MD5Sum. That works fine.
Then i retrieved the image-info (getInfo) from smugmug and noted the md5sum.
Then i used the java messagedigest for MD5 and calculated the md5sum. The result is exactly the same as in point (2.).
However, when i provide this same calculated MD5 sum during the upload, i get an error message from smugmug that the MD5 sum is incorrect... :scratch

This is the code called just before uploading imgData.

final ByteArrayOutputStream bos = new ByteArrayOutputStream();
final BufferedInputStream bufIS = new BufferedInputStream(mData.mIS);
final BufferedOutputStream bufOS = new BufferedOutputStream(bos);
synchronized(smBuffer) {
  int didRead = bufIS.read(smBuffer);
  while (didRead > 0) {
    bufOS.write(smBuffer, 0, didRead); 
    totalWritten += didRead;
 
    didRead = bufIS.read(smBuffer);
  }
  bufOS.flush();
  imgData = bos.toByteArray();
 
  try {
    final MessageDigest md5 = MessageDigest.getInstance("MD5");
    final byte[] digest = md5.digest(imgData);
    for (byte bt : digest) {
      int intBt = ((int)bt) & (0x000000ff);
      [B]md5Sum[/B] += Integer.toHexString(intBt);
    }
  } 
  catch (NoSuchAlgorithmException e) {
    [B]md5Sum[/B] = "";
  }
}

The MD5 sum for image http://www.streetsofboston.com/gallery/3827114/1#221257930 is
Content-Length = 56342
Content-MD5 = 8d57d127da7259e01ef9b92544b93c0

These are the values assigned to the HTTP Put upload as headers.

What am i doing wrong?

This is the error message:
wrong format (ByteCount given: , received. MD5Sum given: , actual.)

(why are the actual values not given in the error-message?)

devbobo · November 17, 2007

Dutchie,

can u post the raw http put request ?

Cheers,

David

flyingdutchie · November 17, 2007

devbobo wrote:

Dutchie,

can u post the raw http put request ?

Cheers,

David

Wow! Talk about a quick reply! Thanks!

Here is the raw data (PUT and response):

PUT /hack/json/1.2.0//C%3A%5Cme.jpg HTTP/1.1
X-Smug-FileName: C:\me.jpg
X-Smug-ResponseType: JSON
Content-Length: 56342
Content-MD5: 8d57d127da7259e01ef9b92544b93c0
X-Smug-SessionID: 83524a08e7203afd4f28f32eea85d4a6
X-Smug-Version: 1.2.0
X-Smug-AlbumID: 3827114
X-Smug-Caption: Test Image
User-Agent: Jakarta Commons-HttpClient/3.1-rc1
Host: upload.smugmug.com
ÿØÿà JFIF  H H  ÿá ÌExif  MM *                  b       j(       1       
 
...
...
 
@aíR6Hms 2&#338; &#8364;gä&#8217;&#8221;ë[É&#8226;
Ï ;&#8240;íÉ&#8230;' àAî¢tÚ<4ü
HTTP/1.1 200 OK
Date: Sat, 17 Nov 2007 22:00:58 GMT
Server: Apache
X-Powered-By: smugmug/1.2.0
Set-Cookie: SMSESS=83524a08e7203afd4f28f32eea85d4a6; path=/; domain=.smugmug.com
Cache-Control: private, max-age=1, must-revalidate
Pragma: 
Content-Length: 107
X-Cnection: close
Content-Type: text/html; charset=utf-8
{"stat":"fail","code":4,"message":"wrong format (ByteCount given: ,  received.  MD5Sum given: ,  actual.)"}

(i removed all the gibberish binary data).

And if Content-MD5: 8d57d127da7259e01ef9b92544b93c0 is not sent, all goes fine.

luke_church · November 18, 2007

flyingdutchie wrote:

Wow! Talk about a quick reply! Thanks!
And if Content-MD5: 8d57d127da7259e01ef9b92544b93c0 is not sent, all goes fine.

I fought with something similar to this a year or so ago. Forgive me if my memory is flaky, I might be barking up the wrong tree.

If I remember correctly there are issues with making sure that the padding on hex strings is done correctly, otherwise they get rejected.

In C# the chunk of code that I have used is (straight out of my source code repository, shout if it doesn't make any sense)

int byteCount = System.Convert.ToInt32(file.Length); //Get byte count for image
string md5sum = null;
//Compute the MD5 Hash for the image
using (FileStream fs = new FileStream(file.FullName, FileMode.Open, FileAccess.Read))
{
//Compute the hash
MD5 md5 = new MD5CryptoServiceProvider();
byte[] hash = md5.ComputeHash(fs);
//Convert it to the format that smugmug want (0 padded bytes, lower case)
StringBuilder buff = new StringBuilder();
foreach (byte hashByte in hash)
{
buff.Append(String.Format("{0:x2}", hashByte));
}
md5sum = buff.ToString();
} //Dispose of the file stream

The Java string formatting APIs will have something similar.

I remember that it wasn't exactly fun finding out what the issue was. Hopefully this will speed you on your way.

Luke

flyingdutchie · November 18, 2007

luke_church wrote:

I fought with something similar to this a year or so ago. Forgive me if my memory is flaky, I might be barking up the wrong tree.

If I remember correctly there are issues with making sure that the padding on hex strings is done correctly, otherwise they get rejected.

In C# the chunk of code that I have used is (straight out of my source code repository, shout if it doesn't make any sense)

int byteCount = System.Convert.ToInt32(file.Length); //Get byte count for image
string md5sum = null;
//Compute the MD5 Hash for the image
using (FileStream fs = new FileStream(file.FullName, FileMode.Open, FileAccess.Read))
{
//Compute the hash
MD5 md5 = new MD5CryptoServiceProvider();
byte[] hash = md5.ComputeHash(fs);
//Convert it to the format that smugmug want (0 padded bytes, lower case)
StringBuilder buff = new StringBuilder();
foreach (byte hashByte in hash)
{
buff.Append(String.Format("{0:x2}", hashByte));
}
md5sum = buff.ToString();
} //Dispose of the file stream

The Java string formatting APIs will have something similar.

I remember that it wasn't exactly fun finding out what the issue was. Hopefully this will speed you on your way.

Luke

Thank you Luke.

Comparing your code to mine, i'm doing the same thing. I get 16 bytes (signed bytes), i make them unsigned and convert each of them in to 2-character hex-digits. The final result is exactly the same as the value returned by Smugmug when i do a 'images.getInfo' on the image.

-- Anton

devbobo · November 18, 2007

Anton,

Try this out, I have a FF extension for uploading testing.

It generates the MD5 hash, so you can compared to Riyad's java api.

Let me know how you get on.

David

flyingdutchie · November 18, 2007

devbobo wrote:

Anton,

Try this out, I have a FF extension for uploading testing.

It generates the MD5 hash, so you can compared to Riyad's java api.

Let me know how you get on.

David

I tried it, but when i run it, i get this error:

XML Parsing Error: not well-formed
Location: chrome://smuguploadtester/content/uploader.xul
Line Number 78, Column 62:       <radio label="api.smugmug.com" value="api.smugmug.com"selected="true" />
---------------------------------------------------------------------------------------^

devbobo · November 18, 2007

urgh sorry...try again

flyingdutchie · November 18, 2007

devbobo wrote:

urgh sorry...try again

It runs now, but the page is entirely transparent... i can only see the borders of the fields and the field-groups. All the backgrounds are transparent...

I'll try anyway and see if i can get it to work.

flyingdutchie · November 18, 2007

flyingdutchie wrote:

It runs now, but the page of the plugin is entirely transparent... i can only see the borders of the fields and the field-groups. All the backgrounds are transparent...

I'll try anyway and see if i can get it to work.

I got it to work... luckily i could click on the input-fields and type in stuff.
However, all the transparent areas are click-through. Instead of clicking on the upload form, my desktop gets focus. Therefore i can not copy (to the clipboard) from the plug-in page.

However, the 'Request' part on the right side shows me the MD5-Sum. It is the exactly the same as i got: 8d57 etc etc... 03c0

The raw request data looks different, though:

POST instead of PUT
xmlrawadd.mg instead of /hack/jason/1.2.0
filename is me.jpg instead of C:\me.jpg
Host is api.smugmug.com instead of upload.smugmug.com

But Content-Length and Content-MD5 are exactly the same.

devbobo · November 18, 2007

flyingdutchie wrote:

PUT /hack/json/1.2.0//C%3A%5Cme.jpg HTTP/1.1
X-Smug-FileName: C:\me.jpg
....

Anton,

this could be the problem...both the PUT location and FileName, should only have the file not that path...

so it should be...

[code]
PUT /me.jpg HTTP/1.1
X-Smug-FileName: me.jpg
....
[/quote]

Give that a go.

Cheers,

David

flyingdutchie · November 18, 2007

devbobo wrote:

Anton,

this could be the problem...both the PUT location and FileName, should only have the file not that path...

so it should be...

[code]
PUT /me.jpg HTTP/1.1
X-Smug-FileName: me.jpg
....

Give that a go.

Cheers,

David[/quote]

Alas, that did not work. I removed the "C:\" in front of the file-name and still it did not work.

BTW: Could the error-message be changed and actual specify what MD5-sum is expected?

Thanks!

devbobo · November 18, 2007

Anton,

please email me the file you're trying to upload to david at [smugmug].

also, grab the latest version of the upload tester, i have fixed the transparent window issue.

David

flyingdutchie · November 19, 2007

devbobo wrote:

Anton,

please email me the file you're trying to upload to david at [smugmug].

also, grab the latest version of the upload tester, i have fixed the transparent window issue.

David

Hi David,

I ran the upload tester and here is a screen shot of the result. The md5-sum is the same. If the md5-sum is the same, what then is different?

PS: I'd like to attach a screen-shot, but DGrin is reporting that i already have "uploaded the maximum of 0 files"

mjohnsonperl · November 20, 2007

flyingdutchie wrote:

This is the error message:
wrong format (ByteCount given: , received. MD5Sum given: , actual.)

You can get this exact response if the content-length isn't calculated correctly. If the content-length is less then what it should be the HTTP PUT will finish early, and the HTTP call with think everything is fine, but when SmugMug's servers try and analyze an incomplete file this kind of response is given.

Here is my REST response I got when testing this.

<rsp stat="fail">
    <err code="4" msg="wrong format (ByteCount given: ,  received.  MD5Sum given: ,  actual.)"/>
</rsp>

My HTTP Request should have been this:

                 '_request' => bless( {
                                        '_content' => sub { "DUMMY" },
                                        '_uri' => bless( do{\(my $o = 'http://upload.smugmug.com/IMG_8589_color%20corrected.jpg')}, 'URI::http' ),
                                        '_headers' => bless( {
                                                               'user-agent' => 'SinkMug/0.1 libwww-perl/5.806',
                                                               'x-smug-version' => '1.2.0',
                                                               'x-smug-responsetype' => 'REST',
                                                               'x-smug-sessionid' => 'b7127dad5252d85001a8b5e5237bbe76',
                                                               'content-length' => 2725682,
                                                               'x-smug-filename' => 'IMG_8589_color corrected.jpg',
                                                               'x-smug-albumid' => '3859440',
                                                               'content-md5' => '7c01e55d7a9f25d0389e864a471eda0f'
                                                             }, 'HTTP::Headers' ),
                                        '_method' => 'PUT'
                                      }, 'HTTP::Request' )

The above request resulted in a successful upload.

I modified my HTTP Request to this:

                 '_request' => bless( {
                                        '_content' => sub { "DUMMY" },
                                        '_uri' => bless( do{\(my $o = 'http://upload.smugmug.com/IMG_8589_color%20corrected.jpg')}, 'URI::http' ),
                                        '_headers' => bless( {
                                                               'user-agent' => 'SinkMug/0.1 libwww-perl/5.806',
                                                               'x-smug-version' => '1.2.0',
                                                               'x-smug-responsetype' => 'REST',
                                                               'x-smug-sessionid' => '4954d1bac20bb0db16c16442ec00d0e2',
                                                               'content-length' => '2725687',
                                                               'x-smug-filename' => 'IMG_8589_color corrected.jpg',
                                                               'x-smug-albumid' => '3859440',
                                                               'content-md5' => '7c01e55d7a9f25d0389e864a471eda0f'
                                                             }, 'HTTP::Headers' ),
                                        '_method' => 'PUT'
                                      }, 'HTTP::Request' )

The only thing I modified was subtracting 5 from the content-length.

I tried adding 5 to the content-length but my script died with an error because the file it was reading to send wasn't big enough or something.

I know I beat my head on this one for a bit last month, but in my case I was opening the image in ASCII mode instead of BINARY mode to calculate the file size to small.

flyingdutchie · November 20, 2007

This means that you should always subtract 5 from the byte-array?

Content-Length = byte_array.length - 5 ?

rkalla · November 20, 2007

flyingdutchie wrote:

This means that you should always subtract 5 from the byte-array?

Content-Length = byte_array.length - 5 ?

Guys, sorry for the delay in responding, I finally got around to testing this and it worked fine using straight API call into the kallasoft SmugMug Java API images.UploadHTTPPut method.

The result, is a very suave-looking Anton

http://kalla.smugmug.com/gallery/3403336#223617633

flyingdutchie · November 20, 2007

rkalla wrote:

Guys, sorry for the delay in responding, I finally got around to testing this and it worked fine using straight API call into the kallasoft SmugMug Java API images.UploadHTTPPut method.

The result, is a very suave-looking Anton
http://kalla.smugmug.com/gallery/3403336#223617633

I'm using the images.UploadHTTPPut method and it works great

.... except that the MD5Sum is not accepted. Riyad, how did you get it to work with the MD5Sum?

PS. That image is from a b-day party of a friend of mine where i dressed up as a bouncer... it was pretty effective, since i almost turned a few people away... hehehehe

rkalla · November 20, 2007

flyingdutchie wrote:

I'm using the images.UploadHTTPPut method and it works great .... except that the MD5Sum is not accepted. Riyad, how did you get it to work with the MD5Sum?

I updated the implementation of UploadHTTPPut to now offer a slew of convenience methods that auto-fill out the HTTP header values for you, so you don't need to pass that stuff anymore. So for example passing in a session ID, album ID and File for an image, the API will load the image, get it's content length, calculate the MD5 hash for it, and set all the remaining headers then do the upload for you.

Additionally you can set all the geo-coding values too if you want. Overall it's just a lot friendly. BUT that being said, I didn't change any of the MD5 logic or anything like that... it just worked. Maybe the whole MD5 issue was a red herring before? Not sure. Either way it should be smoother with the new class.

I'll do another pre-M2 build for you and get it to you later.

flyingdutchie · November 20, 2007

Thanks Riyad!

I sent you an IM a little while ago with the text below. I don't know if you received it or not on your IM:

In SmugFig, you can upload an images as follows:
Image image;
java.io.InputStream myInputStream = new FileInputStream("C:\\somefile.jpg");
...
image.read(myInputStream); // sets the MD5Sum, Size and ImageData
image.setAlbumID(albumID);
image.setFilename("somefile.jpg");
image.setCaption("New Image");
image.update();

image.getMD5Sum() is used to uniquely identify the raw image-data. If the md5sum changed, the image-data has changed.
With your API changes, I hope i still can pass a byte-array and have access to the MD5sum of that byte-array.

rkalla · November 21, 2007

flyingdutchie wrote:

Thanks Riyad!

I sent you an IM a little while ago with the text below. I don't know if you received it or not on your IM:

Ahhh... so you were computing the MD5 independently in your API? I think this might explain why we were seeing a discrepency in the behavior.

In that case the changes I made won't help you, they are automated use-and-throw-away changes done with additional convenience methods... if your API still needs the MD5 (as a hash for example) then you'll still need to compute it yourself.

If it helps, I'm using the Apache, Commons-Codec project's DigesterUtils class, and the md5Hex method:

md5Sum = DigestUtils.md5Hex(imageData);

Maybe we could use the same method call to computer the MD5 so the APIs match up?

flyingdutchie · November 21, 2007

rkalla wrote:
Ahhh... so you were computing the MD5 independently in your API? I think this might explain why we were seeing a discrepency in the behavior.

In that case the changes I made won't help you, they are automated use-and-throw-away changes done with additional convenience methods... if your API still needs the MD5 (as a hash for example) then you'll still need to compute it yourself.

If it helps, I'm using the Apache, Commons-Codec project's DigesterUtils class, and the md5Hex method:
md5Sum = DigestUtils.md5Hex(imageData);
Maybe we could use the same method call to computer the MD5 so the APIs match up?

Hi Riyad,
I'll try that one (DigestUtils) when i get home. I have doubts it would help, because the MD5Sum that i calculate using my current code (MessageDigest) is the same one that is returned by 'images.getInfo' and the same one that is calculated by David's utility (http://hacks.introversion.com.au/SmugUploadTester)

David, did you figure out why my upload fails when i specify an MD5-sum?

devbobo · November 21, 2007

flyingdutchie wrote:

David, did you figure out why my upload fails when i specify an MD5-sum?

Anton,

I forwarded a copy of that image onto Riyad, so that he could verify whether or not he was seeing the same result.

I haven't heard back from him.

David

rkalla · November 21, 2007

devbobo wrote:

Anton,

I forwarded a copy of that image onto Riyad, so that he could verify whether or not he was seeing the same result.

I haven't heard back from him.

David

Per David's request, here's my raw HTTP Put data:

PUT /services/api/json/1.2.1//221257930-O.jpg HTTP/1.1
Content-Length: 56342
Content-MD5: 8d57d127da7259e01ef9b92544b903c0
X-Smug-SessionID: 59c590f38a9ba4c4e636490fdbaba59b
X-Smug-Version: 1.2.1
X-Smug-ResponseType: JSON
X-Smug-AlbumID: 3403336
X-Smug-FileName: 221257930-O.jpg
X-Smug-Caption: [UploadedImage] Caption 1195677657343
X-Smug-Keywords: [UploadedImage] Keyword1 Keyword2 Keyword3
X-Smug-Latitude: 110.56
X-Smug-Longitude: 32.7
X-Smug-Altitude: 787.6
User-Agent: Jakarta Commons-HttpClient/3.1-rc1
Host: upload.smugmug.com

flyingdutchie · November 21, 2007

rkalla wrote:

Per David's request, here's my raw HTTP Put data:

PUT /services/api/json/1.2.1//221257930-O.jpg HTTP/1.1
Content-Length: 56342
Content-MD5: 8d57d127da7259e01ef9b92544b903c0
X-Smug-SessionID: 59c590f38a9ba4c4e636490fdbaba59b
X-Smug-Version: 1.2.1
X-Smug-ResponseType: JSON
X-Smug-AlbumID: 3403336
X-Smug-FileName: 221257930-O.jpg
X-Smug-Caption: [UploadedImage] Caption 1195677657343
X-Smug-Keywords: [UploadedImage] Keyword1 Keyword2 Keyword3
X-Smug-Latitude: 110.56
X-Smug-Longitude: 32.7
X-Smug-Altitude: 787.6
User-Agent: Jakarta Commons-HttpClient/3.1-rc1
Host: upload.smugmug.com

Thanks!

"8d57d127da7259e01ef9b92544b903c0" is the same MD5 my code is sending. Content-Length is the same too "56342".

The main difference is that i uploaded my image using "PUT /services/api/json/1.2.0" and not through "PUT /services/api/json/1.2.1" and X-Smug-Version is set to "1.2.0" instead of "1.2.1".

devbobo · November 21, 2007

flyingdutchie wrote:

Thanks!

"8d57d127da7259e01ef9b92544b903c0" is the same MD5 my code is sending. Content-Length is the same too "56342".

The main difference is that i uploaded my image using "PUT /services/api/json/1.2.0" and not through "PUT /services/api/json/1.2.1" and X-Smug-Version is set to "1.2.0" instead of "1.2.1".

Anton,

putting to /services/api/json/1.2.0 is not required, /filename.jpg is fine. And I don't believe there are any code differences between 1.2.0 and 1.2.1

flyingdutchie · November 21, 2007

devbobo wrote:

Anton,

putting to /services/api/json/1.2.0 is not required, /filename.jpg is fine. And I don't believe there are any code differences between 1.2.0 and 1.2.1

Thanks

Then i don't know what the difference is. The MD5Sum that my code sends is correct, so why is the Smugmug server responding that the MD5 sum is incorrect? (see the raw request-data in one of my earlier messages)

devbobo · November 21, 2007

flyingdutchie wrote:

Thanks

Then i don't know what the difference is. The MD5Sum that my code sends is correct, so why is the Smugmug server responding that the MD5 sum is incorrect? (see the raw request-data in one of my earlier messages)

Anton,

Are you able to provide me with your app so I can do some testing ?

And if need be, do some debugging on internal servers ?

flyingdutchie · November 21, 2007

devbobo wrote:

Anton,

Are you able to provide me with your app so I can do some testing ?

And if need be, do some debugging on internal servers ?

I sent you an e-mail with all the code.

flyingdutchie · November 21, 2007

All the time it was my fault!!

David, Riyad and everyone else: So sorry to have taken up your time. Thank you very much for all your help.

My routine that translates a byte-array into a hex-string, skipped leading 0s.

My MD5Sum
8d57d127da7259e01ef9b92544b93c0

Actual MD5Sum
8d57d127da7259e01ef9b92544b903c0

Again, thanks!

rkalla · November 21, 2007

flyingdutchie wrote:

All the time it was my fault!!

David, Riyad and everyone else: So sorry to have taken up your time. Thank you very much for all your help.

My routine that translates a byte-array into a hex-string, skipped leading 0s.

My MD5Sum
8d57d127da7259e01ef9b92544b93c0

Actual MD5Sum
8d57d127da7259e01ef9b92544b903c0

Again, thanks!

Anton no worries, I'm glad it's working. And I think this thread will be helpful to anyone else that wanders by running into the same problem, atleast they will know where to look right away.

Upload Image: MD5Sum problem

Comments