Advancing open source analysis and development
|
FOSSology Advancing open source analysis and development |
This shows you the differences between the selected revision and the current version of the page.
| task_list 2010/07/29 17:36 | task_list 2010/09/01 02:21 current | ||
|---|---|---|---|
| Line 3: | Line 3: | ||
| ===== V 1.3 ===== | ===== V 1.3 ===== | ||
| //This list represents what we would like to do for v 1.3. We will be working on v 2.0 at the same time as 1.3. Our goal is to get all the high priority items done and probably all the "easy" items. The low priorities will get done if we have time (we are switching to time based releases). It's possible that we will get to none of the low priority items in 1.3. Whatever we don't do will become high priority in 1.4// | //This list represents what we would like to do for v 1.3. We will be working on v 2.0 at the same time as 1.3. Our goal is to get all the high priority items done and probably all the "easy" items. The low priorities will get done if we have time (we are switching to time based releases). It's possible that we will get to none of the low priority items in 1.3. Whatever we don't do will become high priority in 1.4// | ||
| + | ==== 1.3 High ==== | ||
| - | ==== 1.3 high ==== | + | - [[difference_tool|Difference tool]]. This allows one to focus on the differences between two file trees. For example, one might want to compare two versions of a package and focus only on the differences. The definition of a difference depends on the implemented subtraction filters. |
| - | + | - [[perms|Groups.]] Implementing user groups inside of fossology is an indirect but critical requirement for 1.3 because tagging (the real 1.3 requirement) is dependent on having groups to administer tag permissions. | |
| - | - Diff packages (and distros, archives, ...) [[task:Integrate DistroDB (distro package reports)]] shows some of this. | + | - [[tagging | File tagging]] |
| - | - Tagging. For example, tag individual files or containers. If container then the tag might or might not cascade to all it's files. A bucket could tag files. For example, it could tag all the gplv3 packages as "Need Review". Then as they are reviewed, the reviewer would change that tag. This means tags are editable. | + | - The ability to attach a tag (short (max 32 character) tag, plus a long text) to a file or container. |
| - | - To implement tags like this (editable) we need user groups. So write access could be restricted to the group that created the tag, for example. Other ideas can be found here [[tagging | File tags]] (moved from 1.2) | + | - Tagging requires permission for creating, attaching, and viewing tags. Access control at this level does not currently exist in fossology which is why we have a dependency on [[perms|groups and a permission]] strategy. |
| - | - [[task:copyright_author_detection_agent|Improve(replace)the copyright agent]]. A quick experiment showed that we can get better results with simple heuristics rather then the current naive Bayes. Development on branches/new_copyright. | + | - [[task:copyright_author_detection_agent|Replace the copyright agent]]. A quick experiment showed that we can get better results with simple heuristics rather then the current naive Bayes. Development is on branches/new_copyright. |
| - | - [[unit_testing_with_cunit|C code Unit Test]] and Coverage suite. Initial proposal, ideas or framework for how to use C code unit tests and C code coverage to improve our code quality. | + | - fix [[task:cp2foss authentication error]] that prevents bucket agent from getting scheduled. |
| + | - Increase software Quality. This task involves a number of subactivites. | ||
| + | - [[unit_testing_with_cunit_proposal|C code Unit Test]] and Coverage suite. Initial proposal, ideas or framework for how to use C code unit tests and C code coverage to improve our code quality. | ||
| + | - [[PHPUnit Test Framework]] Investigate the feasibility of using this suite of tools forunit testing PHP, performing code coverage analysis of php code and test automation. | ||
| + | - [[Code Coverage for C code]] Investigate and produce a process that allows the capture of code coverage data for our C code using LCOV and GCOV. | ||
| + | - Separate text in UI to facilitate [[i18N]] for Stefan. | ||
| ==== 1.3 easy ==== | ==== 1.3 easy ==== | ||
| - Scheduler.conf does not get created correctly for a cluster. fosscp_agent and fo_notify should only be run on the scheduler host (localhost). Don't forget to include an entry for selftest, too. | - Scheduler.conf does not get created correctly for a cluster. fosscp_agent and fo_notify should only be run on the scheduler host (localhost). Don't forget to include an entry for selftest, too. | ||
| - [[nfs_performance_investigation|NFS I/O performance]] investigate and improve. NFS file I/O is the largest bottleneck for agents, so what can we do to make it faster? Besides faster, making repo access more robust, easier to admin, and take less disk space would be big improvements. | - [[nfs_performance_investigation|NFS I/O performance]] investigate and improve. NFS file I/O is the largest bottleneck for agents, so what can we do to make it faster? Besides faster, making repo access more robust, easier to admin, and take less disk space would be big improvements. | ||
| - | |||
| ==== 1.3 low ==== | ==== 1.3 low ==== | ||
| - [[task:Add/Modify licenses on-the-fly]] this would include a new permission level so that user can correct license and bucket data (other data as well). Perhaps this would update the real data record, but an audit trail could be kept of any changes. (reqt from sutula) | - [[task:Add/Modify licenses on-the-fly]] this would include a new permission level so that user can correct license and bucket data (other data as well). Perhaps this would update the real data record, but an audit trail could be kept of any changes. (reqt from sutula) | ||
| - Improve the unpack agent. The unpack agent used by fossology extracts files from containers. A container is any kind of file that stores other files. For example, a ZIP file contains an archive of different files. Other types of containers include tar, ar, ISO, and rpm files. Look [[1.0.0:agents#unpack | here]] for a full description of unpack agent. What’s wrong with the current unpack agent?: | - Improve the unpack agent. The unpack agent used by fossology extracts files from containers. A container is any kind of file that stores other files. For example, a ZIP file contains an archive of different files. Other types of containers include tar, ar, ISO, and rpm files. Look [[1.0.0:agents#unpack | here]] for a full description of unpack agent. What’s wrong with the current unpack agent?: | ||
| - | - The agent is SLOW. It can take days to unpack a Linux distro. Since Linux distros are of primary interest to the OSRB, fossology needs to be able to unpack distros in hours or minutes, not days. How can we take advantage of multiple CPUs (with the –m switch?) and agent systems to improve performance. __Larry is investigating this issue, and have done some jobs, the performance is improved.__ [[Unpack performance]]. | + | - The agent is SLOW. It can take days to unpack a Linux distro. Since Linux distros are of primary interest to the OSRB, fossology needs to be able to unpack distros in hours or minutes, not days. How can we take advantage of multiple CPUs (with the –m switch?) and agent systems to improve performance. **DONE**: [[Unpack performance]]. |
| - | - The current unpack agent cannot process some Microsoft proprietary formatted files (for example, .msi files). There are windows based command/utilities capable of doing this. Do any exist for use on Linux? __Larry will investigate this issue after unpack performance job.__ [[Unpack Microsoft proprietary formatted files]]. | + | - The current unpack agent cannot process some Microsoft proprietary formatted files. Look into how we do this on a Linux server. **DONE** [[unpack_microsoft_proprietary_formatted_files]] |
| - | - Information/error messages are unhelpful, non-existent and difficult to find in the log file. Log meaningful messages with names of file being processed (if applicable) to a log file for a specific upload – NOT the general fossology.log file. | + | - Since p7zip supports multiple threads and unpacks more than just microsoft install files, look into if it makes sense to replace some of the other utilities we use. ** In Progress** [[replace some of the other utilities used]] |
| + | - The first step to this is to look at our install dependencies for ununpack. Then from that list, which could we remove if we used p7zip (since we already have a dependency on p7zip). So start by making a list of the ununpack dependencies. | ||
| + | - Are there libraries we can use instead the fork/exec'ing each unpacking utility? For all the commands we use, using libraries instead of the executables is something that could potentially, significantly, speed up ununpack. [[instead fork/exec with libraries]] ** in progress ** | ||
| - Deprecate (bSAM) license analyzer and licterms. Either remove entirely or move to its own, unsupported, package. | - Deprecate (bSAM) license analyzer and licterms. Either remove entirely or move to its own, unsupported, package. | ||
| + | - Information/error messages are unhelpful, non-existent and difficult to find in the log file. Log meaningful messages with names of file being processed (if applicable) to a log file for a specific upload – NOT the general fossology.log file. **Pending** [[Perfect the message and log organization]] | ||
| - UI for bucket definition and management (new, change, delete) Not sure where this goes in the priorities. | - UI for bucket definition and management (new, change, delete) Not sure where this goes in the priorities. | ||
| - UI cleanup. Work on inconsistencies and ease of use. Some problems are: | - UI cleanup. Work on inconsistencies and ease of use. Some problems are: | ||
| Line 30: | Line 38: | ||
| - [[http://fossology.org/~bobg/1.2reqs/browsefile.html|Display license changes by package version]] (moved from 1.2) | - [[http://fossology.org/~bobg/1.2reqs/browsefile.html|Display license changes by package version]] (moved from 1.2) | ||
| - Display license differences on a per file basis between versions of any archive (rpm, tar, etc) (moved from 1.2). This includes [[Distro reports]] | - Display license differences on a per file basis between versions of any archive (rpm, tar, etc) (moved from 1.2). This includes [[Distro reports]] | ||
| - | - Browse by folders. Do union of sql query of all uploads in a folder. | + | - Browse by collections. A collection could be all the uploads in a folder, or it could be a user selected set of uploadtrees. To impliment do an sql union of the collection, then proceed normally. This applies to all browsers (browse, nomos, buckets, copyright). |
| - Identify binary packages and the source package they came from (Scott Lamons). The issue here is that the source may not be in the same upload as the binary. So when looking at a binary we need to have an option to choose a source and look at its scans. | - Identify binary packages and the source package they came from (Scott Lamons). The issue here is that the source may not be in the same upload as the binary. So when looking at a binary we need to have an option to choose a source and look at its scans. | ||
| ===== 1.3 cutline ===== | ===== 1.3 cutline ===== | ||
| + | This list is not in priority order. | ||
| + | - Remove copyrights from nomos scan. | ||
| + | - Add to nomos licenses used by ninka and spdx. This includes exceptions. | ||
| + | - Add to nomos, zend license. | ||
| + | - Populate the license_ref table and ship it. Then nomos no longer needs to update it and it will have actual useful information. | ||
| - From slamons: "We need a way to allow users to easily set up new accounts. It would be especially nice if they could log in using their HP email and NT password (or better yet, SiteMinder [[task:implement_single_sign-on|single sign-on session]]). As it is, it is not at all obvious that you need to set yourself up a new account before you start running analysis." | - From slamons: "We need a way to allow users to easily set up new accounts. It would be especially nice if they could log in using their HP email and NT password (or better yet, SiteMinder [[task:implement_single_sign-on|single sign-on session]]). As it is, it is not at all obvious that you need to set yourself up a new account before you start running analysis." | ||
| - Integrated error information. Our current method of logging EVERYTHING to fossology.log makes it difficult to debug issues and view log messages/errors for a particular upload or file. | - Integrated error information. Our current method of logging EVERYTHING to fossology.log makes it difficult to debug issues and view log messages/errors for a particular upload or file. | ||
| Line 41: | Line 54: | ||
| - Remove pfile.pfile_liccount from schema and code (common/common-license.php, plugins/agent-license.php, plugins/agent-license-reanalyze.php, plugins/ui-license.php. This was an experimental feature that mistakenly had code checked in around it. | - Remove pfile.pfile_liccount from schema and code (common/common-license.php, plugins/agent-license.php, plugins/agent-license-reanalyze.php, plugins/ui-license.php. This was an experimental feature that mistakenly had code checked in around it. | ||
| - delagent needs to be more robust. Much of the delagent db updates should probably be done with cascading deletes on the upload. Perhaps cleaning the filesystem should be a separate agent that could be done on a periodic basis? Because of the concurrency problem of deleting unused files from the repo while another agent it adding them, delagent should never run concurrently with unpack or probably any other agent. | - delagent needs to be more robust. Much of the delagent db updates should probably be done with cascading deletes on the upload. Perhaps cleaning the filesystem should be a separate agent that could be done on a periodic basis? Because of the concurrency problem of deleting unused files from the repo while another agent it adding them, delagent should never run concurrently with unpack or probably any other agent. | ||
| + | - Need agent to remove orphan files from the repository. This can happen on a delagent failure. | ||
| - Add license from kernel object modules (license from modsym) to license_file | - Add license from kernel object modules (license from modsym) to license_file | ||
| - New "Compare" checkboxes to compare different files/directories/packages/... | - New "Compare" checkboxes to compare different files/directories/packages/... | ||
| Line 51: | Line 65: | ||
| ===== v 2.0 ===== | ===== v 2.0 ===== | ||
| - | - New scheduler | + | - [[new_scheduler|New scheduler]] |
| - Modular plugins and agents. There are many advantages to [[separate_package | separate optional package]]. Support for optional plugin installs. | - Modular plugins and agents. There are many advantages to [[separate_package | separate optional package]]. Support for optional plugin installs. | ||
| - Improved multihost configuration and installation. | - Improved multihost configuration and installation. | ||
| + | - Consider changing repo layout to work with "large" perhaps 16MB files kind of like hadoop. Using small files like now (avg size ~32k) makes repo deletes take forever and makes copying the repo take forever. For our repo, we could pack as many files as will fit wholly fit into ~16MB. Larger files would be saved as individual files. The whole point is to consolidate the tiny files to use fewer inodes and thus speed up delagent and repo copies. | ||
| + | |||
| ===== UI Mockups ===== | ===== UI Mockups ===== | ||
| All the mockups can be found [[http://fossology.org/~bobg/1.2reqs/|here]]. Or just click on the individual mockups below: | All the mockups can be found [[http://fossology.org/~bobg/1.2reqs/|here]]. Or just click on the individual mockups below: | ||
| Line 79: | Line 95: | ||
| **High priority** - //within the next two releases// | **High priority** - //within the next two releases// | ||
| + | * Password reset capability. (from ScottL). | ||
| * (Mary/Bob) Improve postgres error checking and error reporting | * (Mary/Bob) Improve postgres error checking and error reporting | ||
| * (Mark) improve [[reporting of UI test results]]. | * (Mark) improve [[reporting of UI test results]]. | ||
| - | * (Bob) [[task:write comment plugin | Attachments]] [[http://fossology.org/~bobg/1.2reqs/attachments.html|mockup]] | + | * (Bob) [[task:write comment plugin | Attachments]] The ability to attach an object (text, property, etc..) to a file or container. [[http://fossology.org/~bobg/1.2reqs/attachments.html|mockup]] |
| * (Alex) New machine learning license analyzer (based on sentence clustering). Currently we are calling this [[F1]]. If there are results from both this analysis and fo_nomos, the results will be combined for reporting. Part of this is to compare with Ninka. | * (Alex) New machine learning license analyzer (based on sentence clustering). Currently we are calling this [[F1]]. If there are results from both this analysis and fo_nomos, the results will be combined for reporting. Part of this is to compare with Ninka. | ||
| * [[task:Rest| REST API]] - also look at [[http://fossology.org/pipermail/fossology/2009-December/001489.html | thread started by Phil Martin]] the info in this thread needs to be added to [[task:Rest| REST API]] | * [[task:Rest| REST API]] - also look at [[http://fossology.org/pipermail/fossology/2009-December/001489.html | thread started by Phil Martin]] the info in this thread needs to be added to [[task:Rest| REST API]] | ||
