On Thursday 04 March 2010 04:17:53 Hugo Etchegoyen wrote:
> Charles,
>
> Thank you very much for answering. In the meantime I discovered there is
> a yaffs list and I've subscribed to it, so if you prefer to continue
> this thread in the list just advise.
>
> It looks that you are already addressing my issues, so you can count me
> as a guinea-pig. My timescale for this project is 2-3 months.
>
> To be a little bit more precise about my application, I will try to give
> you some timing figures. They are not very precise because my test board
> is not working right know so I have to count on my memory, but I think
> you'll get the idea.
>
> I have some 200 millisecs available for shutdown. I can save state and
> sync yaffs in less than 100 millisecs if GC doesn't interfere, but when
> it does I can easily run out of time. I did some profiling and realised
> that the execution time of yaffs_GarbageCollectBlock() could climb to
> some 300 millisecs, depending on how many chunks were copied at a time,
> so I came up with the idea of a smaller value for maxCopies. I guess
> your proposed code is safer than just setting maxCopies to 3.
300ms is excessive for gc. I will have a look to see if there is something
interfering.
>
> Background GC would be great since it would improve the average writing
> speed, as most writes would not involve in-line GC. However, since the
> probability of entering GC at power fail would be much smaller, but not
> zero, I think it would still be necessary to decrease maxCopies in order
> to cover the worst case.
Yes indeed, background gc would not block for long.
> Your idea of updating directory times in
> background is also very nice. My application doesn't care about file or
> directory times, so I will mount yaffs with "noatime" and "nodiratime"
> in order to increase performance and flash endurance.
yaffs does not support atimes because it is expensive to do a header write for
eacyaffs is already noatime and nodiratime whether you want it or not :-).
The directory times I'm talking about are the directory mtime and ctime that
get changed every time the directory is changed so that client apps can know
to refresh their view etc. Every time a file is created, renamed or unlinked
we write a directory entry reflect the change. This doubles the number of
writes assocaited with creating or deleting empty files.
The new approach will put the "dirty" directories into a dirty directory list
and use the background thread to periodically (every second or so) update the
directory times. That should reduce the number of directory update writes
significantly.
This is particularly bad when running test code which, say, creates 1000 files
in a directory then deletes them resulting in 2000 directory update writes
(ie. 4000 writes in total), half of which only last for a very brief time.
With the new code there would only be a handful of directory update writes.
>
> I agree that refreshing old data looks much better than refreshing
> random blocks and at the same time does static wear leveling, so as soon
> as you have code for this (or any hints) I will gladly try it.
It is in cvs. Give it a go.
>
> Cheers,
> Hugo
>
> Charles Manning escribió:
> > Hugo
> >
> > This is interesting as I am looking at implementing background gc and
> > revising the gc policies. What are yourr timescales? I'd be interested in
> > someone being a bleeding edge guinea-pig.
> >
> > On Tuesday 02 March 2010 09:41:45 Hugo Etchegoyen wrote:
> >> Dear Charles,
> >>
> >> I intend to use yaffs2 in a uCLinux embedded application, but I need
> >> some tuning.
> >>
> >> My hardware detects power failure and gives me a short time for
> >> shutdown.
> >
> > How short is short?
> >
> >> During this time I must save state to flash. This involves a
> >> few small writes totalling much less that the size of an erase unit.
> >> Then I sync (and unmount?) the yaffs partition.
> >>
> >> As there is little time available for shutdown, I need to make some
> >> modifications to garbage collection:
> >>
> >> 1. I must be able to disable GC during power fail so that it will not
> >> waste valuable time.
> >> 2. GC must be more spreadout than it is now, i.e.
> >> yaffs_GarbageCollectBlock() should only copy a few blocks at a time,
> >> whether aggressive or not, so that power fail will not have to wait too
> >> much before disabling GC.
> >>
> >> As an additional requirement, my flash will be pretty full. It will have
> >> lots of static data and the dynamic data will be very hot, which is a
> >> bad case for endurance. For that reason I also need to:
> >>
> >> 3. Add static GC to yaffs.
> >>
> >> I think (1) can be easily satisfied by raising a flag on power fail and
> >> having yaffs_CheckGarbageCollection() check the flag and do nothing if
> >> it is set.
> >
> > At present it is quite easy to defeat gc by just setting dev->isDoingGC
> > to 1.
> >
> >> Regarding (2), I have already tried a simple modification changing line
> >> 3036 of yaffs_guts.c, in function yaffs_GarbageCollectBlock() as
> >> follows:
> >>
> >> Original: maxCopies = (wholeBlock) ? dev->nChunksPerBlock : 10;
> >> New version: maxCopies = 3;
> >>
> >> I've tried this under rather heavy load and everything seems to work
> >> properly. I hope a figure of 3 guarantees that we will not run out of
> >> free blocks, because during GC three chunks are freed for each new chunk
> >> written.
> >
> > There are some conditions where this policy might fail, but I would
> > expect 3 to probably work.
> >
> > Perhaps
> > maxCopies = (wholeBlock) ? 5 : 3 or something like that would be safer.
> >
> >> Regarding (3), I am thinking of modifying
> >> yaffs_FindBlockForGarbageCollection() so that once in a while, let's say
> >> one time out of 100, it will just choose a random full block. This
> >> should tend to move static data around.
> >
> > It is potentially dangerous to just randomly gc a block unless you also
> > check it is not disqualified.
> >
> > The new block refreshing code will also do what you're wanting. It
> > infrequently gc's the oldest block thus rewriting static data.
> >
> >> I'm aware that all the intended modifications tend to postpone or slow
> >> down GC. Do you think it would be advisable to increase the value of
> >> nReservedBlocks in the yaffs device to compensate, and by how much?
> >
> > nReserved blocks of 5 is already pretty conservative. The dangers are:
> > a) The possibility of a block going bad during gc means we need to keep
> > some extra handy.
> > b) With the current gc logic, nErasedBlocks should never drop below 4,
> > maybe 3. maxcopies = 3 might make things a bit worse because a gc can
> > sometimes end up with very little reclaim (no reclaim is a possibility).
> > That might mean we lose one block of headroom.
> >
> > Thus from a quick mental audit, it sounds like the current level is safe.
> >
> > I might be inclined to do something like:
> > if(dev->nErasedBlocks < 3)
> > maxCopies = dev->nChunksPerBlock; /* Should never happen. This is just a
> > safety net if erased space get really low.*/
> > else if(wholeBlock)
> > maxCopies = 5;
> > else
> > maxCopies = 3;
> >
> > There is currently an ugliness caused by "shrink headers" which constrain
> > the gc order meaning that often we can't collect the blocks that would
> > give the most free space. Sometimes these end up in "chains" meaning
> > there is a sequence of these gc's which causes gc to grind hard.
> >
> > As I mentioned at the top, I am revising gc in two ways:
> > 1) Making it less "lumpy" by doing more smaller gcs.
> > 2) Adding a background thread to do gc behind the scenes meaning that it
> > should only very seldom need to be done in the writing thread. The
> > background thread will also do directory time updating which should cut
> > down on the number of writes when creating files etc.
> >
> > The background gc will target:
> > 1) Trying to keep at least 25% of space in erased blocks. That means
> > writes should happen fast.
> > 2) In-write gc will only happen if erased space gets v low.
> > 3) Eliminate or reduce those pesky shrink header chains so that if gc
> > does happen in the write thread it doesn't become too onerous.
> >
> > -- Charles
(text/plain)