AT&T Labs Research — Leading Invention, Driving Innovation

A relational dependency transform is introduced as a way to exploit information redundancy in conditioning data in a relational database for better compressibility. An optimum relational dependency transform of the relational database is first computed. Fields of the relational database are then sorted topologically based on a weighted, directed graph having nodes representing predictor and predictee fields. For each predictee field in the topological order, a transformed field is then computed via the relationship between predictor and predictee in the optimum relational dependency transform.

The present invention provides efficient window partitioning algorithms for entropy-encoding. The present invention enhances compression performance of entropy encoding based on the approach of modeling a dataset with the frequencies of its n-grams. The present invention may then employ approximation algorithms to compute good partitions in time O(s log s) and O(s) respectively, for any data segment S with length s.

A preload library partitions certain files into segments and transforms the data of the files on a segment by segment basis, each independently of the other. The transformed segments are then stored as part of a transformed file, with each segment corresponding to a bag containing a chunk (of transformed data) and a gap which permits the chunks to grow when data of the segment is modified during a write operation. The bags are followed by a chunk map which contains metadata defining the compressed version of each segment. The preload library intercepts system calls from applications to the OC specifying access to specific segments of a transformed file. The preload library identifies the chunks corresponding to the specified segments, retrieves and reverses the transform on those chunks and passes the data to the application requesting it. For a system call to write data to certain segments, the data is modified and re-transformed by the library before being written back to disk. A preload library provides improved random access into transformed (e.g. compressed, encrypted, etc.) files.

The present invention provides efficient window partitioning algorithms for entropy-encoding. The present invention enhances compression performance of entropy encoding based on the approach of modeling a dataset with the frequencies of its n-grams. The present invention may then employ approximation algorithms to compute good partitions in time O(s log s) and O(s) respectively, for any data segment S with length s.

A method for computing matching windows for delta compression. The method computes pairs of matching source and target data segments irrespective of target data segment size or file offset. The method includes (1) representing a large source data file by a sequence of fixed-size segments; (2) computing a signature for each data segment using its contents such that, with a strong likelihood, two segments are the same if their signatures match; (3) parsing target data using a prefix matching method on such a sequence of signatures of source data to compute matching sequences of segments; and (4) merging closely matched segments as necessary to form matching windows.

A preload library partitions certain files into segments and transforms the data of the files on a segment by segment basis, each independently of the other. The transformed segments are then stored as part of a transformed file, with each segment corresponding to a bag containing a chunk (of transformed data) and a gap which permits the chunks to grow when data of the segment is modified during a write operation. The bags are followed by a chunk map which contains metadata defining the compressed version of each segment. The preload library intercepts system calls from applications to the OC specifying access to specific segments of a transformed file. The preload library identifies the chunks corresponding to the specified segments, retrieves and reverses the transform on those chunks and passes the data to the application requesting it. For a system call to write data to certain segments, the data is modified and re-transformed by the library before being written back to disk. A preload library provides improved random access into transformed (e.g. compressed, encrypted, etc.) files.

The present invention significantly improves the performance of matching target file window partitions to source file window partitions in delta compression. In particular, the present invention enhances delta compression performance of delta compressors based on the idea of modeling a dataset with the frequencies of its n-grams and employs a method to compute good source and target window matching in linear time.

The present invention provides efficient window partitioning algorithms for entropy-encoding. The present invention enhances compression performance of entropy encoding based on the approach of modeling a dataset with the frequencies of its n-grams. The present invention may then employ approximation algorithms to compute good partitions in time O(s*log s) and O(s) respectively, for any data segment S with length s.

The present invention increases the efficiency of performing longest prefix matching operations by selecting a radix-encoded trie structure optimized with respect to memory cost. The structure is optimized by determining memory costs for retrie structures indexed on different numbers of high-order characters, and then selecting the structure corresponding to the lowest memory cost. The optimization improves performance in IP look-up operations as well as longest-prefix matching operations performed on general alphabets.

The present invention is a method and system for buffering and replaying a segment of a voice or data transmission. A connection is established between two end nodes in the network. At an intermediate node in the network between the end nodes, a predetermined segment of the data is maintained in a memory as a buffer. Upon receiving a request from one of the end nodes to replay the segment, that segment is replayed and transmitted to the requesting node. The request may be made by a user at one of the end nodes, or may be made automatically after detecting corrupt data. Because the buffering module is in the network, it need not be incorporated in premises or mobile equipment. Because the buffering module is remote from the requesting end node, the replayed data does not contain errors introduced in the original data between the buffering module and the requesting end node.

The apparent speed of a connection between a browser at a user station and a proxy or gateway on a network such as the Internet is increased by providing a local proxy at the user station which interacts with a remote proxy. While the remote proxy is retrieving a newly requested World Wide Web page, for example, from the appropriate content provider, it may also be sending to the local proxy a stale cached version of that page. When the new version of the page is finally retrieved, the remote proxy determines the differences between the new version and the stale version, and, assuming the differences do not exceed the new page in size, sends the differences to the local proxy which then reconstructs the new page from the differences and the stale version. The local proxy delivers the new page to the browser, which need not even be aware that a local proxy exists; it is aware only that it received the page it requested. Because computational speed and power are frequently higher and cheaper than transmission speed, the apparent speed of the connection between the user station and the network has been increased at modest cost.

The apparent speed of a connection between a browser at a user station and a proxy or gateway on a network such as the Internet is increased by providing a local proxy at the user station which interacts with a remote proxy. While the remote proxy is retrieving a newly requested World Wide Web page, for example, from the appropriate content provider, it may also be sending to the local proxy a stale cached version of that page. When the new version of the page is finally retrieved, the remote proxy determines the differences between the new version and the stale version, and, assuming the differences do not exceed the new page in size, sends the differences to the local proxy which then reconstructs the new page from the differences and the stale version. The local proxy delivers the new page to the browser, which need not even be aware that a local proxy exists; it is aware only that it received the page it requested. Because computational speed and power are frequently higher and cheaper than transmission speed, the apparent speed of the connection between the user station and the network has been increased at modest cost.

The present invention relates to systems for sharing idle workstation computers that are connected together through a network and shared file system. More particularly, a user of a local host workstation may submit jobs for execution on remote workstations. The systems of the present invention select a remote host that is idle in accordance with a decentralized scheduling scheme and then continuously monitor the activity of the remote host on which the job is executing. If the system detects certain activity on the remote host by one of the remote host's primary users, the execution of the job is immediately suspended to prevent inconvenience to the primary users. The system also suspends job execution if the remote host's load average gets too high. Either way, the suspended job is migrated by selecting another idle remote workstation to resume execution of the suspended job (from the point in time at which the last checkpoint occurred).

This patent exploits the observation that any shared secret data can be used to securely exchange new secret data. In particular, if the shared secret data is a file or program that requires frequent updates, an older version can be used to construct a secret key to encrypt a new version with. In this way, there is no need to create and maintain separate passwords for such updates, hence the Passwordless in the title. In addition, files and programs often change little between versions. So protocols are given to use data differencing to both reduce the amount of exchanged data as well as enhancing security. An interesting subcase is the idea of a self-updating program which automatically detects if a new version has arrived, verifies that it is good, then updates itself securely.

When a machine is connected to the Internet via a slow network, the cumulative latency to communicate over the Internet to World Wide Web servers and then transfer documents over the slow network can be significant. We built a system that optimistically transfers data that may be out of date, then sends either a subsequent confirmation that the data is current or a delta to change the older version to the current one. In addition, if both sides of the slow link already store the same older version, just the delta need be transferred to update it. Our mechanism is optimistic because it assumes that much of the time there will be sufficient idle time to transfer most or all of the older version before the newer version is available, and because it assumes that the changes between the two versions will be small relative to the actual document. Timings of retrievals of random URLs in the Internet support the former assumption, while experiments using a version repository of Web documents bear out the latter one. Performance measurements of the optimistic delta system demonstrate that deltas significantly reduce latency when both sides cache the old version, and optimistic deltas can reduce latency, to a lesser degree, when content-provider service times are in the range of seconds or longer.