How to verify NK.BIN builds for QA?

Hello,

Is it possible to verify that two nk.bin files built with identical
source and project configurations are equivalent?

Our QA department needs to build our OS nk.bin from project source we
provide and verify it against a reference nk.bin we provide.

The nk.bin binary image changes between subsequent project builds with
identical source and project settings, so a simple binary comparison
or crc on the images won't work.

Is there a mechanism already present in Platform Builder or perhaps a
tool to achieve this?

Thanks for any suggestions,

-Randy

Valter
Fri Mar 28 01:06:14 PDT 2008

surfbytes@gmail.com wrote in
news:d4da73e5-b4c0-44bc-ae33-6fe7499659d3@u10g2000prn.googlegroups.co
m:

> Hello,
>
> Is it possible to verify that two nk.bin files built with
> identical source and project configurations are equivalent?
>
> Our QA department needs to build our OS nk.bin from project source
> we provide and verify it against a reference nk.bin we provide.
>
> The nk.bin binary image changes between subsequent project builds
> with identical source and project settings, so a simple binary
> comparison or crc on the images won't work.
>
> Is there a mechanism already present in Platform Builder or
> perhaps a tool to achieve this?
>
> Thanks for any suggestions,

You can open and compare two .bin files inside Platform Builder.
You can use the "file\open" command to open the first one (you've to
choose "Windows CE image" as fle type) and then use the button inside
the file view window to open the second one and compare them.


--
Valter Minute
www.fortechembeddedlabs.it
Training, support and development for Windows CE
(the reply address of this message is invalid)

Bruce
Fri Mar 28 05:34:07 PDT 2008

Compare yes, but I am pretty sure that they will be different.

This is an ongoing theme especially in areas like medical, voting machines
and military. I don't know of any real solution.

In this case, the question is why? If it is a question of verifying the
source matches the binary, why not implement a beter build system? If
engineering tags the code in the VCS, and a build engineer who could be in
your QA dept builds for release that could satisfy the requirement.

--
Bruce Eitman (eMVP)
Senior Engineer
beitman AT applieddata DOT net

Applied Data Systems
www.applieddata.net
An ISO 9001:2000 Registered Company
Microsoft WEP Gold-level Member

"Valter Minute" <v_a_l_t_e_r.m_i_n_u_t_e@g_m_a_i_l.com> wrote in message
news:Xns9A6F5C9C4FF97VALTERMINUTE@207.46.248.16...
> surfbytes@gmail.com wrote in
> news:d4da73e5-b4c0-44bc-ae33-6fe7499659d3@u10g2000prn.googlegroups.co
> m:
>
>> Hello,
>>
>> Is it possible to verify that two nk.bin files built with
>> identical source and project configurations are equivalent?
>>
>> Our QA department needs to build our OS nk.bin from project source
>> we provide and verify it against a reference nk.bin we provide.
>>
>> The nk.bin binary image changes between subsequent project builds
>> with identical source and project settings, so a simple binary
>> comparison or crc on the images won't work.
>>
>> Is there a mechanism already present in Platform Builder or
>> perhaps a tool to achieve this?
>>
>> Thanks for any suggestions,
>
> You can open and compare two .bin files inside Platform Builder.
> You can use the "file\open" command to open the first one (you've to
> choose "Windows CE image" as fle type) and then use the button inside
> the file view window to open the second one and compare them.
>
>
> --
> Valter Minute
> www.fortechembeddedlabs.it
> Training, support and development for Windows CE
> (the reply address of this message is invalid)

Valter
Fri Mar 28 06:24:06 PDT 2008

"Bruce Eitman [eMVP]" <beitman.nospam@applieddata.net.nospam> wrote
in news:O4h8VBNkIHA.3780@TK2MSFTNGP06.phx.gbl:

> Compare yes, but I am pretty sure that they will be different.

If you start from the same sources (sam QFEs, compiler version and
everyhing else) I think you should get the same binaries.
The nk.bin can be different because they may contain some signatures
(to recognize a hive registry file, for example), but the exes and
dlls should be the same.

> This is an ongoing theme especially in areas like medical, voting
> machines and military. I don't know of any real solution.
>
> In this case, the question is why? If it is a question of
> verifying the source matches the binary, why not implement a beter
> build system? If engineering tags the code in the VCS, and a
> build engineer who could be in your QA dept builds for release
> that could satisfy the requirement.

I usually got this kind of questions when somebody needs to install PB
on another developer machine or on his new PC... or if they discover
that bugs or features appear/disappear when building on different
machines.... Sometimes problems came from modifications inside the
\PUBLIC subtree, other times from QFEs not installed or not installed
in the same order.
Keeping a machine for "official" builds (those that will be tested by
QA and shipped to customers) is a good rule, and not only for CE,
IMHO.

--
Valter Minute
www.fortechembeddedlabs.it
Training, support and development for Windows CE
(the reply address of this message is invalid)

Bruce
Fri Mar 28 06:52:22 PDT 2008

The binaries are not the same. Date stamps are the biggest problem.

--
Bruce Eitman (eMVP)
Senior Engineer
beitman AT applieddata DOT net

Applied Data Systems
www.applieddata.net
An ISO 9001:2000 Registered Company
Microsoft WEP Gold-level Member

"Valter Minute" <v_a_l_t_e_r.m_i_n_u_t_e@g_m_a_i_l.com> wrote in message
news:Xns9A6F928053646VALTERMINUTE@207.46.248.16...
> "Bruce Eitman [eMVP]" <beitman.nospam@applieddata.net.nospam> wrote
> in news:O4h8VBNkIHA.3780@TK2MSFTNGP06.phx.gbl:
>
>> Compare yes, but I am pretty sure that they will be different.
>
> If you start from the same sources (sam QFEs, compiler version and
> everyhing else) I think you should get the same binaries.
> The nk.bin can be different because they may contain some signatures
> (to recognize a hive registry file, for example), but the exes and
> dlls should be the same.
>
>> This is an ongoing theme especially in areas like medical, voting
>> machines and military. I don't know of any real solution.
>>
>> In this case, the question is why? If it is a question of
>> verifying the source matches the binary, why not implement a beter
>> build system? If engineering tags the code in the VCS, and a
>> build engineer who could be in your QA dept builds for release
>> that could satisfy the requirement.
>
> I usually got this kind of questions when somebody needs to install PB
> on another developer machine or on his new PC... or if they discover
> that bugs or features appear/disappear when building on different
> machines.... Sometimes problems came from modifications inside the
> \PUBLIC subtree, other times from QFEs not installed or not installed
> in the same order.
> Keeping a machine for "official" builds (those that will be tested by
> QA and shipped to customers) is a good rule, and not only for CE,
> IMHO.
>
> --
> Valter Minute
> www.fortechembeddedlabs.it
> Training, support and development for Windows CE
> (the reply address of this message is invalid)

Valter
Fri Mar 28 07:03:49 PDT 2008

"Bruce Eitman [eMVP]" <beitman.nospam@applieddata.net.nospam> wrote in
news:ep8uDtNkIHA.6136@TK2MSFTNGP03.phx.gbl:

> The binaries are not the same. Date stamps are the biggest problem.
>

With binaries you mean the .bin files or each dll and exe inside them?
You are right about dates... I did not consider them.
If it's a matter of being sure that what you are building on PC A is
the same you build on PC B you may compare flatrelease dirs (and any
file included in image from other dirs...).

--
Valter Minute
www.fortechembeddedlabs.it
Training, support and development for Windows CE
(the reply address of this message is invalid)

Bruce
Fri Mar 28 08:11:35 PDT 2008

The compare that you recommend will not tell them if someone modified code
in the Public tree on PC A because the binaries (exes and dlls) are always
different.

--
Bruce Eitman (eMVP)
Senior Engineer
beitman AT applieddata DOT net

Applied Data Systems
www.applieddata.net
An ISO 9001:2000 Registered Company
Microsoft WEP Gold-level Member

"Valter Minute" <v_a_l_t_e_r.m_i_n_u_t_e@g_m_a_i_l.com> wrote in message
news:Xns9A6F993B8EDACVALTERMINUTE@207.46.248.16...
> "Bruce Eitman [eMVP]" <beitman.nospam@applieddata.net.nospam> wrote in
> news:ep8uDtNkIHA.6136@TK2MSFTNGP03.phx.gbl:
>
>> The binaries are not the same. Date stamps are the biggest problem.
>>
>
> With binaries you mean the .bin files or each dll and exe inside them?
> You are right about dates... I did not consider them.
> If it's a matter of being sure that what you are building on PC A is
> the same you build on PC B you may compare flatrelease dirs (and any
> file included in image from other dirs...).
>
> --
> Valter Minute
> www.fortechembeddedlabs.it
> Training, support and development for Windows CE
> (the reply address of this message is invalid)

RJGraham
Fri Mar 28 08:40:41 PDT 2008

On Mar 28, 8:11 am, "Bruce Eitman [eMVP]"
<beitman.nos...@applieddata.net.nospam> wrote:
> The compare that you recommend will not tell them if someone modified code
> in the Public tree on PC A because the binaries (exes and dlls) are always
> different.
>
> --
> Bruce Eitman (eMVP)
> Senior Engineer
> beitman AT applieddata DOT net
>
> Applied Data Systems
> www.applieddata.net
> An ISO 9001:2000 Registered Company
> Microsoft WEP Gold-level Member
>
> "Valter Minute" <v_a_l_t_e_r.m_i_n_u_t_e@g_m_a_i_l.com> wrote in message
>
> news:Xns9A6F993B8EDACVALTERMINUTE@207.46.248.16...
>
> > "Bruce Eitman [eMVP]" <beitman.nos...@applieddata.net.nospam> wrote in
> >news:ep8uDtNkIHA.6136@TK2MSFTNGP03.phx.gbl:
>
> >> The binaries are not the same. Date stamps are the biggest problem.
>
> > With binaries you mean the .bin files or each dll and exe inside them?
> > You are right about dates... I did not consider them.
> > If it's a matter of being sure that what you are building on PC A is
> > the same you build on PC B you may compare flatrelease dirs (and any
> > file included in image from other dirs...).
>
> > --
> > Valter Minute
> >www.fortechembeddedlabs.it
> > Training, support and development for Windows CE
> > (the reply address of this message is invalid)

Thanks to all who have replied.

I think comparing the flatrelease dirs may be the way to go.

I have examined the nk.bin file in some detail, and compared nk.bins
with back to back 'make os image' builds.
Timestamps change of course as do record checksums. Also, some records
have slightly different lengths, which I can only guess is again a
function of timestamping. I have tried 'touching' all files in pre/
post build operations and this reduces the entropy somewhat. But
'makeimg.exe' generates timestamped files while building 'nk.bin'
which can't be 'touched'.

Since we generate crc's for verifying our exe's, QA is expecting the
same for our nk.bin's.

Thanks again,

-Randy

Steve
Fri Mar 28 12:36:31 PDT 2008

> Since we generate crc's for verifying our exe's, QA is expecting the
> same for our nk.bin's.
As Bruce points out the problem here is related to time/date stamps. EXE and
DLLs include stamps in them placed by the linker. Further more every file
(EXE/DLL or data) in the ROM file system has a creation and modified date
for the ROM FS to report up to the OS and applications etc... so these
change the raw binary data, which in turn changes CRCs and that further
changes the binary info. Worse is that all that raw binary changing can
impact the compression algorithms to further propogate changes at the raw
binary level. Any tool to do a real diff would need to take all that into
account and then determine if anything was really different, while ignoring
all the benign changes.

-Steve Maillet

Bruce
Fri Mar 28 12:46:37 PDT 2008

Huh?

So you seem to be saying that even though the binaries change from build to
build, a diff of the release dir tells you something that you think is
enough? Keep in mind the OP isn't talking about just running makimg, he is
talking about getting the code to a different PC and building the whole
thing.

I think that you need to do some more testing before you accept your
findings.

--
Bruce Eitman (eMVP)
Senior Engineer
beitman AT applieddata DOT net

Applied Data Systems
www.applieddata.net
An ISO 9001:2000 Registered Company
Microsoft WEP Gold-level Member

"RJGraham" <surfbytes@gmail.com> wrote in message
news:07c23829-51f5-4c80-ae5d-51c5ea2e58bc@y21g2000hsf.googlegroups.com...
> On Mar 28, 8:11 am, "Bruce Eitman [eMVP]"
> <beitman.nos...@applieddata.net.nospam> wrote:
>> The compare that you recommend will not tell them if someone modified
>> code
>> in the Public tree on PC A because the binaries (exes and dlls) are
>> always
>> different.
>>
>> --
>> Bruce Eitman (eMVP)
>> Senior Engineer
>> beitman AT applieddata DOT net
>>
>> Applied Data Systems
>> www.applieddata.net
>> An ISO 9001:2000 Registered Company
>> Microsoft WEP Gold-level Member
>>
>> "Valter Minute" <v_a_l_t_e_r.m_i_n_u_t_e@g_m_a_i_l.com> wrote in message
>>
>> news:Xns9A6F993B8EDACVALTERMINUTE@207.46.248.16...
>>
>> > "Bruce Eitman [eMVP]" <beitman.nos...@applieddata.net.nospam> wrote in
>> >news:ep8uDtNkIHA.6136@TK2MSFTNGP03.phx.gbl:
>>
>> >> The binaries are not the same. Date stamps are the biggest problem.
>>
>> > With binaries you mean the .bin files or each dll and exe inside them?
>> > You are right about dates... I did not consider them.
>> > If it's a matter of being sure that what you are building on PC A is
>> > the same you build on PC B you may compare flatrelease dirs (and any
>> > file included in image from other dirs...).
>>
>> > --
>> > Valter Minute
>> >www.fortechembeddedlabs.it
>> > Training, support and development for Windows CE
>> > (the reply address of this message is invalid)
>
> Thanks to all who have replied.
>
> I think comparing the flatrelease dirs may be the way to go.
>
> I have examined the nk.bin file in some detail, and compared nk.bins
> with back to back 'make os image' builds.
> Timestamps change of course as do record checksums. Also, some records
> have slightly different lengths, which I can only guess is again a
> function of timestamping. I have tried 'touching' all files in pre/
> post build operations and this reduces the entropy somewhat. But
> 'makeimg.exe' generates timestamped files while building 'nk.bin'
> which can't be 'touched'.
>
> Since we generate crc's for verifying our exe's, QA is expecting the
> same for our nk.bin's.
>
> Thanks again,
>
> -Randy

RJGraham
Fri Mar 28 14:08:41 PDT 2008

On Mar 28, 12:36 pm, "Steve Maillet \(eMVP\)"
<nosp...@entelchyconsulting.com> wrote:
> > Since we generate crc's for verifying our exe's,QAis expecting the
> > same for ournk.bin's.
>
> As Bruce points out the problem here is related to time/date stamps. EXE and
> DLLs include stamps in them placed by the linker. Further more every file
> (EXE/DLL or data) in the ROM file system has a creation and modified date
> for the ROM FS to report up to the OS and applications etc... so these
> change the raw binary data, which in turn changes CRCs and that further
> changes the binary info. Worse is that all that raw binary changing can
> impact the compression algorithms to further propogate changes at the raw
> binary level. Any tool to do a real diff would need to take all that into
> account and then determine if anything was really different, while ignoring
> all the benign changes.
>
> -Steve Maillet

Thanks. It looks like developing an effective diff tool will be more
work than I bargained for.
Just to satisfy my own curiosity though, I am going to take a quick
look at diff'ing the release dirs to see what changes between builds
(aside from file timestamps).


Thanks again,

-Randy
-Randy

Bruce
Fri Mar 28 14:38:05 PDT 2008

You may also want to check out diffing nk.bin files using PB 6.0.

--
Bruce Eitman (eMVP)
Senior Engineer
beitman AT applieddata DOT net

Applied Data Systems
www.applieddata.net
An ISO 9001:2000 Registered Company
Microsoft WEP Gold-level Member

"RJGraham" <surfbytes@gmail.com> wrote in message
news:3f52b5cf-464c-4af2-85e5-611e41ebc908@u69g2000hse.googlegroups.com...
> On Mar 28, 12:36 pm, "Steve Maillet \(eMVP\)"
> <nosp...@entelchyconsulting.com> wrote:
>> > Since we generate crc's for verifying our exe's,QAis expecting the
>> > same for ournk.bin's.
>>
>> As Bruce points out the problem here is related to time/date stamps. EXE
>> and
>> DLLs include stamps in them placed by the linker. Further more every file
>> (EXE/DLL or data) in the ROM file system has a creation and modified date
>> for the ROM FS to report up to the OS and applications etc... so these
>> change the raw binary data, which in turn changes CRCs and that further
>> changes the binary info. Worse is that all that raw binary changing can
>> impact the compression algorithms to further propogate changes at the raw
>> binary level. Any tool to do a real diff would need to take all that into
>> account and then determine if anything was really different, while
>> ignoring
>> all the benign changes.
>>
>> -Steve Maillet
>
> Thanks. It looks like developing an effective diff tool will be more
> work than I bargained for.
> Just to satisfy my own curiosity though, I am going to take a quick
> look at diff'ing the release dirs to see what changes between builds
> (aside from file timestamps).
>
>
> Thanks again,
>
> -Randy
> -Randy