The API library does not support streaming and always reads the full
file into memory.
This workaround copies the signed headers to a Guzzle request and
returns the response as stream.
Makes sure that the paths are trimmed to avoid duplicate entries like
"/test" and "test". This should make this storage slightly faster by
reducing the cache misses.
StoragesService::getStorages() will check the visibility of the backend
and auth mechanism for the storage, and if either are not visible to the
user (aka disabled by admin) then the storage will be filtered out. The
original method StoragesService::getAllStorages() still exists in case
such storages need to be detected, but its use is discouraged.
VisibilityTrait -> PermissionsTrait
PermissionsTrait stores two sets of data, $permissions and
$allowedPermissions (analogous to $visibility and $allowedVisibility of
VisibilityTrait). Each set is a map of user type ('admin' or 'personal')
to permissions (mounting permission, create permission).
The result is that a backend can now be restricted for creation, while
still allowing it to be mounted. This is useful for deprecating backends
or auth mechanisms, preventing new storages being created, while still
allowing existing storages to be mounted.
Introduces the basic password authentication mechanism, along with a
mechanism based on ownCloud credentials stored in the user session.
Change to lib/private is an extension of PermissionsMask, as
isSharable() override was missing.
Session credentials auth mechanism now disables sharing on applied
storages, as credentials will not be available.
Failure to prepare the storage during backend or auth mechanism
manipulation will throw an InsufficientDataForMeaningfulAnswerException,
which is propagated to StorageNotAvailableException in the filesystem
layer via the FailedStorage helper class.
When a storage is unavailable not due to failure, but due to
insufficient data being available, a special 'indeterminate' status is
returned to the configuration UI.
The following functions have been removed:
- addMountPoint()
- removeMountPoint()
- movePersonalMountPoint()
registerBackend() has been rewritten as a shim around BackendService,
allowing legacy code to interact with the new API seamlessly
addMountPoint() was already disconnected from all production code, so
this commit completes the job and removes the function itself, along
with disconnecting and removing related functions. Unit tests have
likewise been removed.
getAbsoluteMountPoints(), getSystemMountPoints() and
getPersonalMountPoints() have been rewritten to use the StoragesServices
Prior to this, the storage class name was stored in mount.json under the
"class" parameter, and the auth mechanism class name under the
"authMechanism" parameter. This decouples the class name from the
identifier used to retrieve the backend or auth mechanism.
Now, backends/auth mechanisms have a unique identifier, which is saved in
the "backend" or "authMechanism" parameter in mount.json respectively.
An identifier is considered unique for the object it references, but the
underlying class may change (e.g. files_external gets pulled into core
and namespaces are modified).
A backend can now specify generic authentication schemes that it
supports, instead of specifying the parameters for its authentication
method directly. This allows multiple authentication mechanisms to be
implemented for a single scheme, providing altered functionality.
This commit introduces the backend framework for this feature, and so at
this point the UI will be broken as the frontend does not specify the
required information.
Terminology:
- authentication scheme
Parameter interface for the authentication method. A backend
supporting the 'password' scheme accepts two parameters, 'user' and
'password'.
- authentication mechanism
Specific mechanism implementing a scheme. Basic mechanisms may
forward configuration options directly to the backend, more advanced
ones may lookup parameters or retrieve them from the session
New dropdown selector for external storage configurations to select the
authentication mechanism to be used.
Authentication mechanisms can have visibilities, just like backends.
The API was extended too to make it easier to add/remove visibilities.
In addition, the concept of 'allowed visibility' has been introduced, so
a backend/auth mechanism can force a maximum visibility level (e.g.
Local storage type) that cannot be overridden by configuration in the
web UI.
An authentication mechanism is a fully instantiated implementation. This
allows an implementation to have dependencies injected into it, e.g. an
\OCP\IDB for database operations.
When a StorageConfig is being prepared for mounting, the authentication
mechanism implementation has manipulateStorage() called,
which inserts the relevant authentication method options into the
storage ready for mounting.
UserGlobalStoragesService reads the global storage configuration,
cherry-picking storages applicable to a user. Writing storages through
this service is forbidden, on punishment of throwing an exception.
Storage IDs may also be config hashes when retrieved from this service,
as it is unable to update the storages with real IDs.
As UserGlobalStoragesService and UserStoragesService share a bit of code
relating to users, that has been split into UserTrait. UserTrait also
allows for the user set to be overridden, rather than using the user
from IUserSession.
Config\ConfigAdapter has been reworked to use UserStoragesService and
UserGlobalStoragesService instead of
OC_Mount_Config::getAbsoluteMountPoints(), further reducing dependance
on that horrible static class.
Backends are registered to the BackendService through new data
structures:
Backends are concrete classes, deriving from
\OCA\Files_External\Lib\Backend\Backend. During construction, the
various configuration parameters of the Backend can be set, in a design
similar to Symfony Console.
DefinitionParameter stores a parameter configuration for an external
storage: name of parameter, human-readable name, type of parameter
(text, password, hidden, checkbox), flags (optional or not).
Storages in the StoragesController now get their parameters validated
server-side (fixes a TODO).
Storage status is saved in the database. Failed storages are rechecked every
10 minutes, while working storages are rechecked every request.
Using the files_external app will recheck all external storages when the
settings page is viewed, or whenever an external storage is saved.
Explicitly clear the stat cache after deleting an empty folder to make
sure it is properly detected as deleted in subsequent requests.
This works around a problem with phpseclib where the folder is properly
deleted remotely but the stat cache was not updated.
The code was missing the "MetadataDirective".
Once added, some other parts of the code failed because the format of mtime was wrong.
So this PR uses the RFC format that the S3 library already uses.
Additionally, the code path where mtime is null was missing. Now defaulting to
the current time.
str_replace for $user substitution was converting the data type of
mountOptions to string. This fix prevents this to happen by making sure
only strings are processed by substitution.
Also added a int conversion when reading the watcher policy
When reading in old mount.json files, they do not contain config ids.
Since these are needed to be able to use the UI and the new service
classes, these will be generated automatically.
The config grouping is based on a config hash.
The mount options are now passed to the UI and stored in a hidden field.
The ext storage controllers and services also know how to load/save them
from the legacy config.
- Added StorageConfig class to replace ugly arrays
- Implemented StorageService and StorageController for Global and User
storages
- Async status checking for storages (from Xenopathic)
- Auto-generate id for external storage configs (not the same as
storage_id)
- Refactor JS classes for external storage settings, this mostly
moves/encapsulated existing global event handlers into the
MountConfigListView class.
- Added some JS unit tests for the external storage UI
Add support for external files accessed via SFTP using public key exchange authentication.
Keys are generated automatically when the configuration is added, or can be regenerated on demand if a key is compromised.
Creation of a new configuration row now triggers focus on that row. This is used to trigger auto-configuration for SFTP keys.
Generated public keys are saved in user's data directory for easy retrieval by an external application.
Add controller for SFTP key generation AJAX
SFTP class initialisation no longer produces a warning if the password field is missing.
Add unit tests for SFTP with key authentication backend
In GDrive, filenames aren't unique, and directories are just
special files - so you can have multiple files with the same
name, multiple directories with the same name, and even files
with the same names as directories.
OC doesn't handle this at all, though, and just wants to act
as if file and directory names *are* unique. So when renaming,
we must check if there's an existing object with the same
file or directory name before we commit the rename, and
explicitly delete it if the rename is successful. (Other
providers like dropbox do the same for files, but intentionally
don't do it for directories; we really need to do it for
directories too.)
A good way to observe this is to run the storage unit tests
and look at the state of the Drive afterwards. Without this
commit, there will be several copies of all the test files
and directories. After this commit, there's just one of each.
We can't just say "hey, Drive lets us do this, what's the
problem?" because we don't handle multiple-objects, same-name
cases - getDriveFile() just bails and prints an error if it
searches for the file or directory with a given name and gets
multiple results.
Sometimes there are bugs that cause setupFS() to be called for
non-existing users. Instead of failing hard and breaking the instance,
this fix simply logs a warning.
opendir() trims the path passed then calls getDriveFile() -
which immediately does the same trim operation. This breaks
opendir() on the root directory, which causes the failure of
the testStat() test when it checks the ctime of the root dir.
ownCloud passes us a Unix time integer, but the GDrive API wants
an RFC3339-formatted date. Actually it wants a single particular
RFC3339 format, not just anything that complies will do - it
requires the fractions to be specified, though RFC3339 doesn't.
This resolves issue #11267 (and was also noted by PVince81 in
reviewing PR #6989).
This is a slightly hacky workaround for
https://github.com/google/google-api-php-client/issues/59 .
There's a bug in the Google library which makes it go nuts on
file uploads and transfer *way* too much data if compression is
enabled and it's using its own IO handler (not curl). Upstream
'fixed' this (by disabling compression) for one upload
mechanism, but not for the one we use. The bug doesn't seem to
happen if the google lib detects that curl is available and
decides to use it instead of its own handler. So, let's disable
compression, but only if it looks like the Google lib's check
for curl is going to fail.
Allow specifying a protocol in the host field when mounting another
ownCloud instance. Note that this was already possible with the WebDAV
config but this bug made it inconsistent.
folder size and mtime is always unknown in s3
more s3 fixes
make rescanDelay of root dir configurable, add on the fly update of legacy storage ids, !isset -> empty when checking strings
reduce number of http calls on remove and rmdir
fix typo
maintain deprecated \OC::$session when getting or setting the session via the server container or UserSession
restore order os OC::$session and OC::$CLI
remove unneded initialization of dummy session
write back session when $useCustomSession is true
log warning when deprecated app is used
As constants not defined within a class cannot be automatically found by the
autoloader moving those constants into a class makes them accessible to
code which uses them.
Signed-off-by: Stephan Peijnik <speijnik@anexia-it.com>
For some reason the aws-sdk-php package does not caclulate the
signiture correctly when accessing an object in a bucket with a name of
'.'.
When we are at the top of a S3 bucket there is a need(?) to have a directory
name. Per standard Unix the name picked was '.' (dot or period). This
choice exercises the aws-sdk bug.
This fix is to add a field to the method to store the name to use instead of
'.' which at this point is hard coded to '<root>'. We also add a private
function 'cleanKey()' which will test for the '.' name and replace it with
the variable. Finally all calls to manipulate objects where the path is
not obviously not '.' are processed through cleanKey().
An example where we don't process through clean key would be
'Key' => $path.'/',
Use correct relationship operator
Per feed back use === instead of ==
use '/' instead of '<root>'
Now the external storage correctly returns the mount points visible only
for the current user by using the method getAbsoluteMountPoints() which
is already filtered.
Since that call was missing the backend name which is important for the
UI, this one was added as well.
Each storage backend has a default priority, assigned to any system mounts
created in ownCloud. mount.json can be manually modified to change these
priorities.
The priority order is as follows:
* Personal
* User
* Group
* Global
Within each mount type, the mount with the highest priority is active.
The storage backend defaults were chosen to be the following:
* Local - 150
* Remote storage - 100
* SMB / CIFS with OC login - 90
An issue existed where `readData` used `OCP\User::getUser()` to get the user
for personal mount points, which worked in all situations apart from when a
personal mount point was used for sharing, so the return from `getUser()` is
not the user that owns the share. As such, any personal mount points would not
work correctly when shared.
`readData` and `writeData` have been changed from using a `$isPersonal`
boolean to using a `$user` string|null. `$isPersonal = false` can now be
written as `$user = NULL` (or left out in the case of `readData`), and
`$isPersonal = true` can be written as `$user = OCP\User::getUser()`.
Dependency messages now appear below the configuration options instead of above
Reworked dependency check method to support consolidated messages for multiple backends
Conflicts:
apps/files_external/lib/google.php
apps/files_external/lib/swift.php
apps/files_external/templates/settings.php
Added check for backend's checkDependencies method to OC_Mount_Config::getBackends() when backend is configured to have some instead of blindly calling it and crashing
Conflicts:
apps/files_external/lib/config.php
apps/files_external/lib/google.php
Moved dependency check to the individual backends instead of the config
Conflicts:
apps/files_external/appinfo/app.php
apps/files_external/lib/config.php
The touch() operation now uses "UpdateMetadata()" instead of "Update()"
which doesn't clear the object's contents.
This fixes syncing, as the sync client needs to use touch to update the
object's mtime.
To check for shares, the code attempts to connect anonymously to the share.
In most cases this will fail with NT_STATUS_ACCESS_DENIED, so the regex array
used for parsing the output of smbclient in smb4php has been overridden to
treat such output as success.
The 'test' method for storage classes can now take a single parameter,
$isPersonal, which allows the storage to adjust the tests performed based on
if they are being configured as personal shares or as system shares.
Shares authenticated with user credentials (aka not hard coded) cannot be
accessed by other users, breaking sharing. This change completely revokes
sharing for such shares
getDisplayName would return the display name of the user, not great if it is a
canonical string. The uid passed back from 'login' is the UUID of the user, so
also not suitable. The login name from the session is the username the user
used to log in to ownCloud in the first place, which is what is needed.
Selecting 'SMB/CIFS Auto' in the mounts configuration allows an SMB/CIFS
mount to be configured that uses the credentials of the user logging in to
authenticate to the server.
Optionally, the username can be used as the share name, permitting home shares
to be dynamically mounted.
Merging of configurations is whenever the same config is available for
multiple users/groups, in which case the config is considered as a
single one by the UI, and shows multiple users/groups selected.
Fixed merging logic to make sure that class, mount point and options are
the same before merging them.
Fixed merging to work correctly when the same mount point path is used
for separate users and configs. These are now correctly shows in the UI
as separate entries.
The old password field is now preserved in the JSON structure to make
sure that the order is preserved.
This is a quick fix until the UI is fixed to not rely on the PHP array
key order.
- test config was written correctly (global and local)
- test that personal mount points can be added for allowed backends
- test that personal mount points cannot be added for disallowed backends
- added $skipTest flag to make it possible add mount points without
doing an actual check/connection (note: this is necessary because the
static class OC_Mount_Config cannot be mocked)
- Some WebDAV servers like lighttpd need paths in URLs to be properly
encoded
- Added error log output when curl connection failed
- Added check for 'resourcetype' in case the WebDAV server doesn't
support/return it
- Fixed touch() to return false if the server doesn't implement
PROPPATCH
- Added optional delay in WebDAV unit tests to use when testing against
lighttpd's WebDAV
Added private var $certPath to store the user root cert
Move logic to determine the $certPath path to the constructor and modify to get the path from OC_User::getHome()
Add curl options to use the certificate to avoid certificate errors with self-signed certicates in the downdload/upload method so we don't get blank files
Some storages need to use different calls for deleting files or folders,
usually unlink() and rmdir().
Fixes#4532 (SMB dir deletion)
Fixes#5941 (FTP dir deletion)
Note that the extra is_dir() should be fast because it's read from the
stat cache.
- fixed touch function to return true on success
- fixed local metadata cache to remove deleted files/directories
- fixed getMetaData() to ignore files reported as deleted by
Dropbox
- fixed "file not found" case to not log as exception
- fixed "overwrite on rename" case
- fixed unit tests to exclude unsupported cases
- added unit test for touch return value
- unit tests for Dropbox should all run correctly now
When uploading files through WebDAV, a part file is created and a rename
operation is performed with the expectation that the part file
overwrites an existing file, if any.
This fix makes the SMB external storage delete the target file before
renaming, as smbclient doesn't support overwrite on move/rename.
Fixes#5348
When an ext storage doesn't implement isReadable(), always returning
true made the file scanner believe that the file exists and creates a
cache entry with the size zero.
This fix makes the default impl of isReadable() use file_exists().
Fixes#5940
To make it possible to use the short ownCloud URL (without specifying
webdav.php or remote.php), a new backend is available for ownCloud.
The user must specify the host + context path in the "Url" field (which
is mapped to the "host" parameter) and the subdir to mount in the "Root"
field.
This is to prevent confusion because some users forget to append
webdav.php or remote.php to the WebDAV URL.
Fixes#4923
Fixes#3466 because the test method of external filesystems uses stat() to detect if the given parameters are okay.
Changes to 3rdparty are necessary as well:
https://github.com/owncloud/3rdparty/pull/33
Robin Appelman
Thomas Müller
Robin McCorkell
Vincent Petry
Daniel Tosello
Tim Dettrick
Scrutinizer Auto-Fixer
Lukas Reschke
Jesus Macias Portela
Jesús Macias
Morris Jobke
Bjoern Schiessle
Martin
Andreas Fischer
Robin McCorkell
Roeland Jago Douma
Frank Karlitschek
Jenkins for ownCloud
SA
Ross Nicoll
Jörn Friedrich Dreyer
Volkan Gezer
Adam Williamson
Robin Appelman
Joas Schilling
André Gaul
Johan Björk
Stephan Peijnik
Christopher T. Johnson
Sascha Schmidt
Lukas Reschke
Philipp Kapfer
Jan-Christoph Borchardt
Frank Karlitschek
Philipp Schmitt
Tigran Mkrtchyan
Michael Monreal
Carlos Cerrillo
Morris Jobke
hkjolhede
Christian Berendt
Lennart Rosam
Arthur Schiwon
Bart Visscher
Georg Ehrke
icewind1991
Kondou
Alexander Bogdanov
Michael Gapczynski